Sep 19 10:34:26.721388 ip-10-0-143-38 systemd[1]: Starting Kubernetes Kubelet... Sep 19 10:34:30.027423 ip-10-0-143-38 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 19 10:34:30.027423 ip-10-0-143-38 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 19 10:34:30.027423 ip-10-0-143-38 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 19 10:34:30.027423 ip-10-0-143-38 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 19 10:34:30.027423 ip-10-0-143-38 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 19 10:34:30.028129 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.027499 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 19 10:34:30.031184 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031168 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 10:34:30.031184 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031182 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 19 10:34:30.031184 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031186 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031190 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031193 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031196 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031198 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031201 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031204 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031207 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031211 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031214 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031216 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031225 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031228 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031231 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031234 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031236 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031239 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031241 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031244 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031247 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 10:34:30.031276 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031250 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031253 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031255 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031258 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031261 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031263 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031266 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031269 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031272 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031275 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031277 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031280 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031283 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031286 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031288 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031291 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031294 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031296 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031299 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031301 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 10:34:30.031754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031304 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031306 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031309 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031311 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031314 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031317 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031319 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031322 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031324 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031329 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031332 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031335 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031338 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031342 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031346 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031349 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031351 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031354 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031357 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 10:34:30.032273 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031360 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031362 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031365 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031369 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031371 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031374 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031377 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031379 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031382 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031386 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031390 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031393 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031396 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031398 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031402 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031405 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031409 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031411 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031414 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 10:34:30.032728 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031418 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031421 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031424 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031426 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031429 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031431 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031434 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031437 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031440 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031443 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031445 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031448 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031451 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031453 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031456 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031459 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031461 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031463 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.031466 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032157 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 10:34:30.033242 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032164 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032168 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032172 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032175 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032178 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032182 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032185 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032188 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032190 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032193 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032195 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032198 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032200 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032205 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032208 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032211 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032214 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032217 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032220 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 10:34:30.033733 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032223 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032227 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032230 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032233 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032236 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032239 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032243 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032245 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032249 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032251 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032256 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032259 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032262 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032265 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032268 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032270 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032273 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032275 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032278 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 10:34:30.034220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032281 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032284 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032286 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032289 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032292 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032294 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032297 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032299 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032302 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032304 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032306 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032309 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032311 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032315 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032318 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032320 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032323 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032326 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032328 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032331 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 10:34:30.034696 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032333 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032336 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032339 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032341 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032344 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032346 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032349 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032351 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032354 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032356 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032359 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032361 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032364 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032367 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032370 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032372 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032375 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032377 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032380 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032382 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 10:34:30.035206 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032384 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032387 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032390 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032393 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032395 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032398 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032401 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032404 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032406 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032409 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032412 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032414 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032417 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032419 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032422 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032424 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032427 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032429 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032432 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032435 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 10:34:30.035702 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032821 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032826 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032830 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032833 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032836 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032840 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032843 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032846 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032851 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032854 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032857 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032859 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032862 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032864 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032867 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032870 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032872 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032875 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032878 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032881 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 10:34:30.036256 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032884 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032887 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032889 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032892 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032894 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032897 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032899 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032902 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032904 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032907 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032910 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032912 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032915 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032917 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032920 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032922 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032925 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032928 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032930 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 10:34:30.036745 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032933 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032935 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032938 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032942 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032945 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032948 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032951 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032953 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032956 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032959 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032962 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032964 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032967 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032969 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032972 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032975 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032979 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032981 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032984 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 10:34:30.037212 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032987 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032990 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032992 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032995 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.032997 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033000 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033002 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033005 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033008 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033010 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033015 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033018 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033021 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033023 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033026 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033031 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033033 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033036 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033038 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033040 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 10:34:30.037676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033044 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033046 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033049 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033051 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033054 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033070 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033073 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033075 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033078 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033080 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033083 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033086 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033088 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033091 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033093 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033096 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033098 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033101 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033104 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033106 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 10:34:30.038204 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033109 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033180 2579 flags.go:64] FLAG: --address="0.0.0.0" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033222 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033227 2579 flags.go:64] FLAG: --anonymous-auth="true" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033232 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033236 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033240 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033245 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033249 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033253 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033256 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033259 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033262 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033265 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033268 2579 flags.go:64] FLAG: --cgroup-root="" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033271 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033275 2579 flags.go:64] FLAG: --client-ca-file="" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033277 2579 flags.go:64] FLAG: --cloud-provider="external" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033280 2579 flags.go:64] FLAG: --cluster-dns="[]" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033284 2579 flags.go:64] FLAG: --cluster-domain="" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033287 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033290 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033293 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033297 2579 flags.go:64] FLAG: --container-log-max-files="5" Sep 19 10:34:30.038691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033300 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033303 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033306 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033310 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033313 2579 flags.go:64] FLAG: --contention-profiling="false" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033315 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033318 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033322 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033325 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033328 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033332 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033335 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033338 2579 flags.go:64] FLAG: --enable-load-reader="false" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033341 2579 flags.go:64] FLAG: --enable-server="true" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033345 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033350 2579 flags.go:64] FLAG: --event-burst="100" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033354 2579 flags.go:64] FLAG: --event-qps="50" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033356 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033359 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033363 2579 flags.go:64] FLAG: --eviction-hard="" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033366 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033369 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033372 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033375 2579 flags.go:64] FLAG: --eviction-soft="" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033378 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 19 10:34:30.039303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033381 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033384 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033387 2579 flags.go:64] FLAG: --experimental-mounter-path="" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033390 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033393 2579 flags.go:64] FLAG: --fail-swap-on="true" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033396 2579 flags.go:64] FLAG: --feature-gates="" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033400 2579 flags.go:64] FLAG: --file-check-frequency="20s" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033402 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033406 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033409 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033412 2579 flags.go:64] FLAG: --healthz-port="10248" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033415 2579 flags.go:64] FLAG: --help="false" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033418 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033421 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033424 2579 flags.go:64] FLAG: --http-check-frequency="20s" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033426 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033429 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033434 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033437 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033440 2579 flags.go:64] FLAG: --image-service-endpoint="" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033442 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033446 2579 flags.go:64] FLAG: --kube-api-burst="100" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033451 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033455 2579 flags.go:64] FLAG: --kube-api-qps="50" Sep 19 10:34:30.039917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033458 2579 flags.go:64] FLAG: --kube-reserved="" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033460 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033463 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033466 2579 flags.go:64] FLAG: --kubelet-cgroups="" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033469 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033472 2579 flags.go:64] FLAG: --lock-file="" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033475 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033478 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033481 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033486 2579 flags.go:64] FLAG: --log-json-split-stream="false" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033488 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033491 2579 flags.go:64] FLAG: --log-text-split-stream="false" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033494 2579 flags.go:64] FLAG: --logging-format="text" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033497 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033500 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033503 2579 flags.go:64] FLAG: --manifest-url="" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033506 2579 flags.go:64] FLAG: --manifest-url-header="" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033510 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033513 2579 flags.go:64] FLAG: --max-open-files="1000000" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033517 2579 flags.go:64] FLAG: --max-pods="110" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033520 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033523 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033526 2579 flags.go:64] FLAG: --memory-manager-policy="None" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033529 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033532 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 19 10:34:30.040570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033536 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033539 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033545 2579 flags.go:64] FLAG: --node-status-max-images="50" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033548 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033552 2579 flags.go:64] FLAG: --oom-score-adj="-999" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033556 2579 flags.go:64] FLAG: --pod-cidr="" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033559 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033564 2579 flags.go:64] FLAG: --pod-manifest-path="" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033567 2579 flags.go:64] FLAG: --pod-max-pids="-1" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033570 2579 flags.go:64] FLAG: --pods-per-core="0" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033573 2579 flags.go:64] FLAG: --port="10250" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033576 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033579 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-067699f12926996e7" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033583 2579 flags.go:64] FLAG: --qos-reserved="" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033586 2579 flags.go:64] FLAG: --read-only-port="10255" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033589 2579 flags.go:64] FLAG: --register-node="true" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033592 2579 flags.go:64] FLAG: --register-with-taints="" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033595 2579 flags.go:64] FLAG: --registry-burst="10" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033598 2579 flags.go:64] FLAG: --registry-qps="5" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033601 2579 flags.go:64] FLAG: --reserved-cpus="" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033604 2579 flags.go:64] FLAG: --reserved-memory="" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033608 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033611 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033614 2579 flags.go:64] FLAG: --rotate-certificates="false" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033617 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 19 10:34:30.041211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033620 2579 flags.go:64] FLAG: --runonce="false" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033622 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033625 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033628 2579 flags.go:64] FLAG: --seccomp-default="false" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033631 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033634 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033637 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033640 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033645 2579 flags.go:64] FLAG: --storage-driver-password="root" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033648 2579 flags.go:64] FLAG: --storage-driver-secure="false" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033650 2579 flags.go:64] FLAG: --storage-driver-table="stats" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033653 2579 flags.go:64] FLAG: --storage-driver-user="root" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033657 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033660 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033664 2579 flags.go:64] FLAG: --system-cgroups="" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033667 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033672 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033675 2579 flags.go:64] FLAG: --tls-cert-file="" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033678 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033682 2579 flags.go:64] FLAG: --tls-min-version="" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033685 2579 flags.go:64] FLAG: --tls-private-key-file="" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033688 2579 flags.go:64] FLAG: --topology-manager-policy="none" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033691 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033694 2579 flags.go:64] FLAG: --topology-manager-scope="container" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033697 2579 flags.go:64] FLAG: --v="2" Sep 19 10:34:30.041845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033700 2579 flags.go:64] FLAG: --version="false" Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033707 2579 flags.go:64] FLAG: --vmodule="" Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033710 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.033714 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033824 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033827 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033831 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033834 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033836 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033839 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033842 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033844 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033847 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033849 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033852 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033856 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033859 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033862 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033864 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033869 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 19 10:34:30.042465 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033872 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033875 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033878 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033880 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033883 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033886 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033888 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033891 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033893 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033896 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033899 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033901 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033904 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033906 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033911 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033914 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033917 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033920 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033922 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033925 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 10:34:30.042961 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033928 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033932 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033936 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033940 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033943 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033946 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033949 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033953 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033956 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033959 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033962 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033965 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033968 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033970 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033973 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033977 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033980 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033982 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033986 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 10:34:30.043476 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033989 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033992 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033995 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.033998 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034000 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034003 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034006 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034008 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034011 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034014 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034016 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034019 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034021 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034024 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034027 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034029 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034032 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034035 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034037 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034040 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 10:34:30.043940 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034044 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034046 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034049 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034052 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034068 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034071 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034073 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034076 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034079 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034430 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034434 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034436 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034439 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034442 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034444 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034447 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034450 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034453 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034456 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034458 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 10:34:30.044453 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034461 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034463 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034466 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.034469 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.034926 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.042039 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.042052 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.042087 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.042094 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042147 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042152 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042156 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042159 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042163 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042166 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 10:34:30.044979 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042169 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042172 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042174 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042177 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042180 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042182 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042185 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042188 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042191 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042193 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042196 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042199 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042202 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042204 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042207 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042210 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042213 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042216 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042219 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042221 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 10:34:30.045390 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042224 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042227 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042230 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042233 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042236 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042239 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042242 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042244 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042247 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042249 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042252 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042256 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042260 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042263 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042266 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042268 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042271 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042273 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042276 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 10:34:30.045922 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042278 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042281 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042284 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042287 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042289 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042292 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042296 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042299 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042302 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042305 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042308 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042310 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042313 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042316 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042318 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042322 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042324 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042328 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042330 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042333 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 10:34:30.046412 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042336 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042338 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042341 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042344 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042346 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042349 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042351 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042354 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042356 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042359 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042361 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042364 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042367 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042369 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042372 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042375 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042378 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042380 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042383 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042385 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 10:34:30.046909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042388 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042390 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042393 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042396 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042398 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042401 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042404 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042407 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042409 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042413 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042417 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042420 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042423 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042426 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 10:34:30.047535 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.042431 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042548 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042553 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042556 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042559 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042563 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042567 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042570 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042573 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042576 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042578 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042581 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042584 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042587 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042590 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042592 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042595 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042597 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042600 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042603 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 10:34:30.047871 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042605 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042608 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042610 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042612 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042616 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042618 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042621 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042624 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042626 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042629 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042631 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042634 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042636 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042640 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042643 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042647 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042649 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042652 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042655 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042658 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 10:34:30.048393 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042661 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042663 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042666 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042668 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042670 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042673 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042676 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042678 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042681 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042683 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042686 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042689 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042691 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042693 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042696 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042698 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042701 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042704 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042707 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 10:34:30.048873 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042710 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042713 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042715 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042718 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042721 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042723 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042726 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042728 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042731 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042733 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042736 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042739 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042741 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042744 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042746 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042749 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042751 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042754 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042756 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042759 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 10:34:30.049347 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042762 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042764 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042767 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042770 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042772 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042775 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042777 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042780 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042782 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042785 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042787 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042790 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042792 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042795 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042797 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042800 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042806 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042809 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042811 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042814 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 10:34:30.049829 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:30.042816 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 10:34:30.050530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.042821 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 19 10:34:30.050530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.042915 2579 server.go:962] "Client rotation is on, will bootstrap in background" Sep 19 10:34:30.050530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.044929 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 19 10:34:30.050530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.046281 2579 server.go:1019] "Starting client certificate rotation" Sep 19 10:34:30.050530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.046383 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 19 10:34:30.050530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.046418 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 19 10:34:30.050530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.048220 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 19 10:34:30.050530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.049994 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 19 10:34:30.057268 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.057250 2579 log.go:25] "Validated CRI v1 runtime API" Sep 19 10:34:30.058817 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.058796 2579 log.go:25] "Validated CRI v1 image API" Sep 19 10:34:30.060143 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.060122 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 19 10:34:30.062825 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.062807 2579 fs.go:135] Filesystem UUIDs: map[1f5833a5-c43c-491a-8e5f-11375033c468:/dev/nvme0n1p4 4dfe7fa1-e6d8-4296-9734-a55ce0f2f099:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 19 10:34:30.062897 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.062823 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 19 10:34:30.067489 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.067384 2579 manager.go:216] Machine: {Timestamp:2026-09-19 10:34:30.06690257 +0000 UTC m=+1.979407276 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112639 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29beac4a60a8cd3f77cafdc75490ac SystemUUID:ec29beac-4a60-a8cd-3f77-cafdc75490ac BootID:415e3168-25f6-4881-88ad-45300f83e016 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:90:a7:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:90:a7:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:8b:d7:b3:40:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 19 10:34:30.067489 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.067486 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 19 10:34:30.067598 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.067583 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 19 10:34:30.068963 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.068942 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 19 10:34:30.069110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.068965 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-38.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 19 10:34:30.070358 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.070348 2579 topology_manager.go:138] "Creating topology manager with none policy" Sep 19 10:34:30.070399 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.070360 2579 container_manager_linux.go:309] "Creating device plugin manager" Sep 19 10:34:30.070399 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.070368 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 19 10:34:30.070399 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.070382 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 19 10:34:30.070478 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.070447 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 19 10:34:30.073575 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.073564 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 19 10:34:30.073679 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.073670 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 19 10:34:30.073716 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.073701 2579 kubelet.go:486] "Attempting to sync node with API server" Sep 19 10:34:30.073716 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.073711 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 19 10:34:30.073771 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.073721 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 19 10:34:30.073771 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.073730 2579 kubelet.go:398] "Adding apiserver pod source" Sep 19 10:34:30.073771 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.073738 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 19 10:34:30.074055 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.074042 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 19 10:34:30.074113 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.074080 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 19 10:34:30.075756 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.075737 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 19 10:34:30.077711 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.077699 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 19 10:34:30.077760 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.077728 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 19 10:34:30.078967 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.078953 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 19 10:34:30.079022 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.078977 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 19 10:34:30.079022 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.078975 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 19 10:34:30.079022 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.078985 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 19 10:34:30.079022 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.078995 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 19 10:34:30.079022 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.079007 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 19 10:34:30.079022 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.079016 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 19 10:34:30.079234 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.079035 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 19 10:34:30.079234 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.079044 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 19 10:34:30.079234 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.079053 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 19 10:34:30.079234 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.079076 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 19 10:34:30.079234 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.079088 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 19 10:34:30.079234 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.079097 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 19 10:34:30.079234 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.079145 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 19 10:34:30.079234 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.079153 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 19 10:34:30.083115 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.083084 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-38.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 19 10:34:30.083115 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.083102 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 19 10:34:30.083264 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.083143 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 19 10:34:30.083264 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.083178 2579 server.go:1268] "Started kubelet" Sep 19 10:34:30.083349 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.083263 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 19 10:34:30.083349 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.083317 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 19 10:34:30.083402 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.083368 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 19 10:34:30.084034 ip-10-0-143-38 systemd[1]: Started Kubernetes Kubelet. Sep 19 10:34:30.090884 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.090864 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 19 10:34:30.091497 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.091485 2579 server.go:310] "Adding debug handlers to kubelet server" Sep 19 10:34:30.116212 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.116188 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 19 10:34:30.120726 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.119820 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-38.ec2.internal.18d6b269f54d72e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-38.ec2.internal,UID:ip-10-0-143-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-38.ec2.internal,},FirstTimestamp:2026-09-19 10:34:30.083154662 +0000 UTC m=+1.995659369,LastTimestamp:2026-09-19 10:34:30.083154662 +0000 UTC m=+1.995659369,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-38.ec2.internal,}" Sep 19 10:34:30.120820 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.120733 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 19 10:34:30.120820 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.120754 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 19 10:34:30.121019 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.120834 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:30.121252 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.121225 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 19 10:34:30.121252 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.121227 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 19 10:34:30.121354 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.121265 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 19 10:34:30.121354 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.121350 2579 reconstruct.go:97] "Volume reconstruction finished" Sep 19 10:34:30.121416 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.121361 2579 reconciler.go:29] "Reconciler: start to sync state" Sep 19 10:34:30.122655 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.122633 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 19 10:34:30.122655 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.122652 2579 factory.go:55] Registering systemd factory Sep 19 10:34:30.122800 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.122660 2579 factory.go:223] Registration of the systemd container factory successfully Sep 19 10:34:30.122885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.122854 2579 factory.go:153] Registering CRI-O factory Sep 19 10:34:30.122885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.122873 2579 factory.go:223] Registration of the crio container factory successfully Sep 19 10:34:30.122990 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.122891 2579 factory.go:103] Registering Raw factory Sep 19 10:34:30.122990 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.122903 2579 manager.go:1196] Started watching for new ooms in manager Sep 19 10:34:30.123321 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.123306 2579 manager.go:349] Starting recovery of all containers Sep 19 10:34:30.124903 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.124876 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5vxt6" Sep 19 10:34:30.132257 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.132233 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5vxt6" Sep 19 10:34:30.132732 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.132718 2579 manager.go:354] Recovery completed Sep 19 10:34:30.133715 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.133684 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 19 10:34:30.133786 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.133708 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 19 10:34:30.136870 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.136856 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 10:34:30.138664 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.138650 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory" Sep 19 10:34:30.138726 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.138675 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 10:34:30.138726 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.138685 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID" Sep 19 10:34:30.139294 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.139278 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 19 10:34:30.139294 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.139292 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 19 10:34:30.139416 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.139309 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 19 10:34:30.139416 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.139315 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 10:34:30.140033 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.140018 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory" Sep 19 10:34:30.140111 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.140042 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 10:34:30.140111 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.140051 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID" Sep 19 10:34:30.140411 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.140401 2579 policy_none.go:49] "None policy: Start" Sep 19 10:34:30.140448 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.140415 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 19 10:34:30.140448 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.140425 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 19 10:34:30.140831 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.140772 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-38.ec2.internal.18d6b269f89c7044 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-38.ec2.internal,UID:ip-10-0-143-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-38.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-38.ec2.internal,},FirstTimestamp:2026-09-19 10:34:30.13866298 +0000 UTC m=+2.051167687,LastTimestamp:2026-09-19 10:34:30.13866298 +0000 UTC m=+2.051167687,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-38.ec2.internal,}" Sep 19 10:34:30.141462 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.141449 2579 policy_none.go:47] "Start" Sep 19 10:34:30.179592 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.179579 2579 manager.go:337] "Starting Device Plugin manager" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.179621 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.179631 2579 server.go:87] "Starting device plugin registration server" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.179909 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.179937 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.180027 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.180113 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.180124 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.181087 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.181133 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.187873 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.188978 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.189032 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.189072 2579 kubelet.go:2448] "Starting kubelet main sync loop" Sep 19 10:34:30.190345 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.189115 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 19 10:34:30.191011 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.190988 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 10:34:30.281128 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.280997 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 10:34:30.283528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.282527 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory" Sep 19 10:34:30.283528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.282564 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 10:34:30.283528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.282581 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID" Sep 19 10:34:30.283528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.282618 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.289764 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.289205 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"] Sep 19 10:34:30.289864 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.289821 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 10:34:30.290632 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.290614 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory" Sep 19 10:34:30.290694 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.290643 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 10:34:30.290694 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.290658 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID" Sep 19 10:34:30.291977 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.291962 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.292018 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.291983 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-38.ec2.internal\": node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:30.292755 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.292743 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 10:34:30.292865 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.292850 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.292927 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.292878 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 10:34:30.294424 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.294401 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory" Sep 19 10:34:30.294424 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.294424 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 10:34:30.294530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.294433 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID" Sep 19 10:34:30.294530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.294409 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory" Sep 19 10:34:30.294530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.294457 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 10:34:30.294530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.294467 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID" Sep 19 10:34:30.296616 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.296599 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.296704 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.296627 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 10:34:30.297362 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.297341 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory" Sep 19 10:34:30.297443 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.297374 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 10:34:30.297443 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.297386 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID" Sep 19 10:34:30.317578 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.317557 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-38.ec2.internal\" not found" node="ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.318017 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.318002 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:30.321987 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.321970 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-38.ec2.internal\" not found" node="ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.418877 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.418854 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:30.422134 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.422118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c827f395adacad1f6ed10de19d147344-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"c827f395adacad1f6ed10de19d147344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.422186 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.422142 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c827f395adacad1f6ed10de19d147344-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"c827f395adacad1f6ed10de19d147344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.422186 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.422158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/54f93cd73487bad8dda8b78673c5c557-config\") pod \"kube-apiserver-proxy-ip-10-0-143-38.ec2.internal\" (UID: \"54f93cd73487bad8dda8b78673c5c557\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.519302 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.519277 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:30.522539 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.522512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c827f395adacad1f6ed10de19d147344-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"c827f395adacad1f6ed10de19d147344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.522631 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.522543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c827f395adacad1f6ed10de19d147344-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"c827f395adacad1f6ed10de19d147344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.522631 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.522560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/54f93cd73487bad8dda8b78673c5c557-config\") pod \"kube-apiserver-proxy-ip-10-0-143-38.ec2.internal\" (UID: \"54f93cd73487bad8dda8b78673c5c557\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.522631 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.522622 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c827f395adacad1f6ed10de19d147344-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"c827f395adacad1f6ed10de19d147344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.522727 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.522649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/54f93cd73487bad8dda8b78673c5c557-config\") pod \"kube-apiserver-proxy-ip-10-0-143-38.ec2.internal\" (UID: \"54f93cd73487bad8dda8b78673c5c557\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.522727 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.522622 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c827f395adacad1f6ed10de19d147344-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"c827f395adacad1f6ed10de19d147344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.619947 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.619878 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:30.620978 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.620965 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.625040 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.625028 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" Sep 19 10:34:30.721017 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.720991 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:30.821345 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.821318 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:30.899349 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:30.899317 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 10:34:30.922251 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:30.922230 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:31.022728 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:31.022703 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:31.046007 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:31.045988 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 19 10:34:31.046382 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:31.046139 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 19 10:34:31.046382 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:31.046139 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 19 10:34:31.123194 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:31.123168 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:31.137467 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:31.137436 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 10:29:30 +0000 UTC" deadline="2028-03-31 17:06:40.624645292 +0000 UTC" Sep 19 10:34:31.137467 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:31.137463 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13422h32m9.487185452s" Sep 19 10:34:31.223722 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:31.223671 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:31.324185 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:31.324155 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found" Sep 19 10:34:31.376673 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:31.376541 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 10:34:31.378473 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:31.378436 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc827f395adacad1f6ed10de19d147344.slice/crio-4d177fa8e1a8e4bf800ea026734f799de25e729c0303c7e6ac099d0221a339b6 WatchSource:0}: Error finding container 4d177fa8e1a8e4bf800ea026734f799de25e729c0303c7e6ac099d0221a339b6: Status 404 returned error can't find the container with id 4d177fa8e1a8e4bf800ea026734f799de25e729c0303c7e6ac099d0221a339b6 Sep 19 10:34:31.379135 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:31.379113 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f93cd73487bad8dda8b78673c5c557.slice/crio-bf061fcab32af57d2ee0a81a5c313d031ebe16e7c90191d4195b83eb5cd7f718 WatchSource:0}: Error finding container bf061fcab32af57d2ee0a81a5c313d031ebe16e7c90191d4195b83eb5cd7f718: Status 404 returned error can't find the container with id bf061fcab32af57d2ee0a81a5c313d031ebe16e7c90191d4195b83eb5cd7f718 Sep 19 10:34:31.384413 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:31.384398 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 19 10:34:31.421393 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:31.421365 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" Sep 19 10:34:31.436236 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:31.436218 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 19 10:34:31.436972 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:31.436961 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" Sep 19 10:34:31.445254 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:31.445239 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 19 10:34:31.590185 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:31.590129 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 10:34:32.075092 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.075042 2579 apiserver.go:52] "Watching apiserver" Sep 19 10:34:32.082712 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.082686 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 19 10:34:32.083162 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.083131 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-hcl5q","kube-system/global-pull-secret-syncer-d2djw","kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz","openshift-multus/multus-additional-cni-plugins-2jrfz","openshift-multus/multus-dpm4k","openshift-multus/network-metrics-daemon-9xvrh","openshift-network-diagnostics/network-check-target-ntfqc","openshift-ovn-kubernetes/ovnkube-node-jzdrv","kube-system/konnectivity-agent-jzzq5","openshift-cluster-node-tuning-operator/tuned-cljtb","openshift-dns/node-resolver-c9c22","openshift-image-registry/node-ca-k2ddk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"] Sep 19 10:34:32.085678 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.085655 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:32.085774 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.085754 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:32.087882 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.087861 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k2ddk" Sep 19 10:34:32.090600 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.090166 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:32.090600 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.090239 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 19 10:34:32.090600 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.090239 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:32.090600 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.090290 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 19 10:34:32.090600 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.090393 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 19 10:34:32.090600 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.090491 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7sn6x\"" Sep 19 10:34:32.094763 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.092912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.095162 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.095133 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-npkq5\"" Sep 19 10:34:32.097179 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.095755 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 19 10:34:32.097179 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.095813 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 19 10:34:32.097179 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.095867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.097179 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.095760 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 19 10:34:32.098516 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.098489 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 19 10:34:32.098726 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.098710 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 19 10:34:32.098878 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.098864 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sh9pb\"" Sep 19 10:34:32.099013 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.098998 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 19 10:34:32.099248 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.099231 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 19 10:34:32.099350 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.099334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.099413 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.099235 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 19 10:34:32.100546 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.100530 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.101503 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.101487 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-n5hzt\"" Sep 19 10:34:32.101596 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.101506 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 19 10:34:32.102752 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.102734 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 19 10:34:32.102980 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.102963 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 19 10:34:32.103039 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.103006 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 19 10:34:32.103288 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.103272 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hcl5q" Sep 19 10:34:32.104131 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.104109 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 19 10:34:32.104131 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.104114 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 19 10:34:32.104319 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.104301 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 19 10:34:32.105359 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.104347 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p6sdr\"" Sep 19 10:34:32.105663 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.105419 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 19 10:34:32.105752 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.105665 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:32.105752 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.105722 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:32.105863 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.105843 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 19 10:34:32.106023 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.105938 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4j4jp\"" Sep 19 10:34:32.106151 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.106031 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 19 10:34:32.108561 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.108543 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jzzq5" Sep 19 10:34:32.110769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.110750 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 19 10:34:32.111114 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.110969 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4srmw\"" Sep 19 10:34:32.111207 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.111166 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 19 10:34:32.111419 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.111403 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.113501 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.113450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c9c22" Sep 19 10:34:32.113824 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.113801 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 19 10:34:32.113963 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.113945 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-clnx7\"" Sep 19 10:34:32.114737 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.114009 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 19 10:34:32.115550 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.115531 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 19 10:34:32.115958 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.115939 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v2rfc\"" Sep 19 10:34:32.116152 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.116136 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 19 10:34:32.121935 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.121100 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 19 10:34:32.122640 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.122621 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130047 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-kubelet-dir\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4345b3c-abd0-4cc2-8b21-a87b10324f78-kubelet-config\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130155 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4345b3c-abd0-4cc2-8b21-a87b10324f78-dbus\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ae16cac-a8e0-4e04-b938-da9258b3a883-host-slash\") pod \"iptables-alerter-hcl5q\" (UID: \"7ae16cac-a8e0-4e04-b938-da9258b3a883\") " pod="openshift-network-operator/iptables-alerter-hcl5q" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-var-lib-kubelet\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-sysctl-d\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/747ac66b-6228-471f-9409-77c6246911dd-serviceca\") pod \"node-ca-k2ddk\" (UID: \"747ac66b-6228-471f-9409-77c6246911dd\") " pod="openshift-image-registry/node-ca-k2ddk" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-registration-dir\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb8bc\" (UniqueName: \"kubernetes.io/projected/5bcf3b15-117f-4aaf-a94a-08997ee640c8-kube-api-access-tb8bc\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8457194-8923-4c07-9e06-b06463781252-ovnkube-script-lib\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/712bf5a4-5979-4e67-bab1-92d6befbf2e8-cni-binary-copy\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-sys\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-system-cni-dir\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-var-lib-cni-bin\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.131263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130505 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npj58\" (UniqueName: \"kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58\") pod \"network-check-target-ntfqc\" (UID: \"65cc3016-6e8a-4a8c-be7b-3634a31508a1\") " pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130562 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bcf3b15-117f-4aaf-a94a-08997ee640c8-cnibin\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-systemd-units\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-run-ovn\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-run-ovn-kubernetes\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-run-openvswitch\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8457194-8923-4c07-9e06-b06463781252-env-overrides\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d60a105e-478f-4a4b-b1b9-daff4de47c93-konnectivity-ca\") pod \"konnectivity-agent-jzzq5\" (UID: \"d60a105e-478f-4a4b-b1b9-daff4de47c93\") " pod="kube-system/konnectivity-agent-jzzq5" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-systemd\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-run\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b32d1889-b166-48b3-979f-8a99b875bb78-etc-tuned\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/747ac66b-6228-471f-9409-77c6246911dd-host\") pod \"node-ca-k2ddk\" (UID: \"747ac66b-6228-471f-9409-77c6246911dd\") " pod="openshift-image-registry/node-ca-k2ddk" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-kubelet\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-cni-netd\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-socket-dir\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.132101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130947 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-multus-cni-dir\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.130970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-multus-socket-dir-parent\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-run-netns\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131027 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-hostroot\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-kubernetes\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-sysctl-conf\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-etc-kubernetes\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b32d1889-b166-48b3-979f-8a99b875bb78-tmp\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e5b260f-bb0a-4096-9a9f-f4a534e77eea-hosts-file\") pod \"node-resolver-c9c22\" (UID: \"9e5b260f-bb0a-4096-9a9f-f4a534e77eea\") " pod="openshift-dns/node-resolver-c9c22" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjw6\" (UniqueName: \"kubernetes.io/projected/1728a47c-9260-477e-8b40-42c3cfdee6db-kube-api-access-drjw6\") pod \"network-metrics-daemon-9xvrh\" (UID: \"1728a47c-9260-477e-8b40-42c3cfdee6db\") " pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-log-socket\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d60a105e-478f-4a4b-b1b9-daff4de47c93-agent-certs\") pod \"konnectivity-agent-jzzq5\" (UID: \"d60a105e-478f-4a4b-b1b9-daff4de47c93\") " pod="kube-system/konnectivity-agent-jzzq5" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131566 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bcf3b15-117f-4aaf-a94a-08997ee640c8-system-cni-dir\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131775 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-cnibin\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-var-lib-cni-multus\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131925 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhp7f\" (UniqueName: \"kubernetes.io/projected/712bf5a4-5979-4e67-bab1-92d6befbf2e8-kube-api-access-jhp7f\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.132951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.131972 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-modprobe-d\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-host\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e5b260f-bb0a-4096-9a9f-f4a534e77eea-tmp-dir\") pod \"node-resolver-c9c22\" (UID: \"9e5b260f-bb0a-4096-9a9f-f4a534e77eea\") " pod="openshift-dns/node-resolver-c9c22" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-device-dir\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132135 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-etc-selinux\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bcf3b15-117f-4aaf-a94a-08997ee640c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bcf3b15-117f-4aaf-a94a-08997ee640c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-run-netns\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132258 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-run-systemd\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd7qg\" (UniqueName: \"kubernetes.io/projected/e8457194-8923-4c07-9e06-b06463781252-kube-api-access-gd7qg\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132308 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-multus-conf-dir\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-run-multus-certs\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-cni-bin\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvwcj\" (UniqueName: \"kubernetes.io/projected/7ae16cac-a8e0-4e04-b938-da9258b3a883-kube-api-access-tvwcj\") pod \"iptables-alerter-hcl5q\" (UID: \"7ae16cac-a8e0-4e04-b938-da9258b3a883\") " pod="openshift-network-operator/iptables-alerter-hcl5q" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132429 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjmh8\" (UniqueName: \"kubernetes.io/projected/b32d1889-b166-48b3-979f-8a99b875bb78-kube-api-access-fjmh8\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bcf3b15-117f-4aaf-a94a-08997ee640c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.133746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132484 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-slash\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-etc-openvswitch\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8457194-8923-4c07-9e06-b06463781252-ovn-node-metrics-cert\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ae16cac-a8e0-4e04-b938-da9258b3a883-iptables-alerter-script\") pod \"iptables-alerter-hcl5q\" (UID: \"7ae16cac-a8e0-4e04-b938-da9258b3a883\") " pod="openshift-network-operator/iptables-alerter-hcl5q" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-run-k8s-cni-cncf-io\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-var-lib-kubelet\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49bgv\" (UniqueName: \"kubernetes.io/projected/9e5b260f-bb0a-4096-9a9f-f4a534e77eea-kube-api-access-49bgv\") pod \"node-resolver-c9c22\" (UID: \"9e5b260f-bb0a-4096-9a9f-f4a534e77eea\") " pod="openshift-dns/node-resolver-c9c22" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132704 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bcf3b15-117f-4aaf-a94a-08997ee640c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-lib-modules\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m62j\" (UniqueName: \"kubernetes.io/projected/35d5cc80-a8a7-4ce0-8195-9464476aae21-kube-api-access-6m62j\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-var-lib-openvswitch\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-node-log\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/712bf5a4-5979-4e67-bab1-92d6befbf2e8-multus-daemon-config\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-sysconfig\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs\") pod \"network-metrics-daemon-9xvrh\" (UID: \"1728a47c-9260-477e-8b40-42c3cfdee6db\") " pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjknj\" (UniqueName: \"kubernetes.io/projected/747ac66b-6228-471f-9409-77c6246911dd-kube-api-access-cjknj\") pod \"node-ca-k2ddk\" (UID: \"747ac66b-6228-471f-9409-77c6246911dd\") " pod="openshift-image-registry/node-ca-k2ddk" Sep 19 10:34:32.134531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-sys-fs\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.135321 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.132971 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8457194-8923-4c07-9e06-b06463781252-ovnkube-config\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.148649 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.148570 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wdvc6" Sep 19 10:34:32.158290 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.158250 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wdvc6" Sep 19 10:34:32.197249 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.196349 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" event={"ID":"c827f395adacad1f6ed10de19d147344","Type":"ContainerStarted","Data":"4d177fa8e1a8e4bf800ea026734f799de25e729c0303c7e6ac099d0221a339b6"} Sep 19 10:34:32.197639 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.197614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" event={"ID":"54f93cd73487bad8dda8b78673c5c557","Type":"ContainerStarted","Data":"bf061fcab32af57d2ee0a81a5c313d031ebe16e7c90191d4195b83eb5cd7f718"} Sep 19 10:34:32.233885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.233855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-cni-bin\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.234025 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.233898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvwcj\" (UniqueName: \"kubernetes.io/projected/7ae16cac-a8e0-4e04-b938-da9258b3a883-kube-api-access-tvwcj\") pod \"iptables-alerter-hcl5q\" (UID: \"7ae16cac-a8e0-4e04-b938-da9258b3a883\") " pod="openshift-network-operator/iptables-alerter-hcl5q" Sep 19 10:34:32.234025 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.233964 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-cni-bin\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.234025 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjmh8\" (UniqueName: \"kubernetes.io/projected/b32d1889-b166-48b3-979f-8a99b875bb78-kube-api-access-fjmh8\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.234220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bcf3b15-117f-4aaf-a94a-08997ee640c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.234220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-slash\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.234220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-etc-openvswitch\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.234220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8457194-8923-4c07-9e06-b06463781252-ovn-node-metrics-cert\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.234220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ae16cac-a8e0-4e04-b938-da9258b3a883-iptables-alerter-script\") pod \"iptables-alerter-hcl5q\" (UID: \"7ae16cac-a8e0-4e04-b938-da9258b3a883\") " pod="openshift-network-operator/iptables-alerter-hcl5q" Sep 19 10:34:32.234220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-run-k8s-cni-cncf-io\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.234407 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-var-lib-kubelet\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.234407 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49bgv\" (UniqueName: \"kubernetes.io/projected/9e5b260f-bb0a-4096-9a9f-f4a534e77eea-kube-api-access-49bgv\") pod \"node-resolver-c9c22\" (UID: \"9e5b260f-bb0a-4096-9a9f-f4a534e77eea\") " pod="openshift-dns/node-resolver-c9c22" Sep 19 10:34:32.234407 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bcf3b15-117f-4aaf-a94a-08997ee640c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.234407 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-lib-modules\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.234407 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m62j\" (UniqueName: \"kubernetes.io/projected/35d5cc80-a8a7-4ce0-8195-9464476aae21-kube-api-access-6m62j\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.234407 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-var-lib-openvswitch\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.234407 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-node-log\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.234407 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/712bf5a4-5979-4e67-bab1-92d6befbf2e8-multus-daemon-config\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.234720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-sysconfig\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.234720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234450 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs\") pod \"network-metrics-daemon-9xvrh\" (UID: \"1728a47c-9260-477e-8b40-42c3cfdee6db\") " pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:32.234720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjknj\" (UniqueName: \"kubernetes.io/projected/747ac66b-6228-471f-9409-77c6246911dd-kube-api-access-cjknj\") pod \"node-ca-k2ddk\" (UID: \"747ac66b-6228-471f-9409-77c6246911dd\") " pod="openshift-image-registry/node-ca-k2ddk" Sep 19 10:34:32.234720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-sys-fs\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.234720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8457194-8923-4c07-9e06-b06463781252-ovnkube-config\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.234720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:32.234720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-kubelet-dir\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.234720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4345b3c-abd0-4cc2-8b21-a87b10324f78-kubelet-config\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:32.234720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4345b3c-abd0-4cc2-8b21-a87b10324f78-dbus\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:32.234720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ae16cac-a8e0-4e04-b938-da9258b3a883-host-slash\") pod \"iptables-alerter-hcl5q\" (UID: \"7ae16cac-a8e0-4e04-b938-da9258b3a883\") " pod="openshift-network-operator/iptables-alerter-hcl5q" Sep 19 10:34:32.234720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-var-lib-kubelet\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.234720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-sysctl-d\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/747ac66b-6228-471f-9409-77c6246911dd-serviceca\") pod \"node-ca-k2ddk\" (UID: \"747ac66b-6228-471f-9409-77c6246911dd\") " pod="openshift-image-registry/node-ca-k2ddk" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-registration-dir\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb8bc\" (UniqueName: \"kubernetes.io/projected/5bcf3b15-117f-4aaf-a94a-08997ee640c8-kube-api-access-tb8bc\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234820 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bcf3b15-117f-4aaf-a94a-08997ee640c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8457194-8923-4c07-9e06-b06463781252-ovnkube-script-lib\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/712bf5a4-5979-4e67-bab1-92d6befbf2e8-cni-binary-copy\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-sys\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-system-cni-dir\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-var-lib-cni-bin\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npj58\" (UniqueName: \"kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58\") pod \"network-check-target-ntfqc\" (UID: \"65cc3016-6e8a-4a8c-be7b-3634a31508a1\") " pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.234979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bcf3b15-117f-4aaf-a94a-08997ee640c8-cnibin\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-systemd-units\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-run-ovn\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-run-ovn-kubernetes\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-run-openvswitch\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8457194-8923-4c07-9e06-b06463781252-env-overrides\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.235240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235152 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d60a105e-478f-4a4b-b1b9-daff4de47c93-konnectivity-ca\") pod \"konnectivity-agent-jzzq5\" (UID: \"d60a105e-478f-4a4b-b1b9-daff4de47c93\") " pod="kube-system/konnectivity-agent-jzzq5" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-systemd\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-run\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b32d1889-b166-48b3-979f-8a99b875bb78-etc-tuned\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/747ac66b-6228-471f-9409-77c6246911dd-host\") pod \"node-ca-k2ddk\" (UID: \"747ac66b-6228-471f-9409-77c6246911dd\") " pod="openshift-image-registry/node-ca-k2ddk" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-kubelet\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-cni-netd\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-socket-dir\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-multus-cni-dir\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-multus-socket-dir-parent\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-run-netns\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-hostroot\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235483 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8457194-8923-4c07-9e06-b06463781252-ovnkube-config\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-kubernetes\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-sysctl-conf\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235826 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-multus-socket-dir-parent\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235864 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-slash\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.236087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.235897 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-etc-openvswitch\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236516 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236552 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/747ac66b-6228-471f-9409-77c6246911dd-host\") pod \"node-ca-k2ddk\" (UID: \"747ac66b-6228-471f-9409-77c6246911dd\") " pod="openshift-image-registry/node-ca-k2ddk" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236604 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-systemd\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236610 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-sys-fs\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-etc-kubernetes\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b32d1889-b166-48b3-979f-8a99b875bb78-tmp\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e5b260f-bb0a-4096-9a9f-f4a534e77eea-hosts-file\") pod \"node-resolver-c9c22\" (UID: \"9e5b260f-bb0a-4096-9a9f-f4a534e77eea\") " pod="openshift-dns/node-resolver-c9c22" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236727 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjw6\" (UniqueName: \"kubernetes.io/projected/1728a47c-9260-477e-8b40-42c3cfdee6db-kube-api-access-drjw6\") pod \"network-metrics-daemon-9xvrh\" (UID: \"1728a47c-9260-477e-8b40-42c3cfdee6db\") " pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-log-socket\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d60a105e-478f-4a4b-b1b9-daff4de47c93-agent-certs\") pod \"konnectivity-agent-jzzq5\" (UID: \"d60a105e-478f-4a4b-b1b9-daff4de47c93\") " pod="kube-system/konnectivity-agent-jzzq5" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bcf3b15-117f-4aaf-a94a-08997ee640c8-system-cni-dir\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236814 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-node-log\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-cnibin\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-var-lib-cni-multus\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhp7f\" (UniqueName: \"kubernetes.io/projected/712bf5a4-5979-4e67-bab1-92d6befbf2e8-kube-api-access-jhp7f\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.236922 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-modprobe-d\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-host\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e5b260f-bb0a-4096-9a9f-f4a534e77eea-tmp-dir\") pod \"node-resolver-c9c22\" (UID: \"9e5b260f-bb0a-4096-9a9f-f4a534e77eea\") " pod="openshift-dns/node-resolver-c9c22" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.236986 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4345b3c-abd0-4cc2-8b21-a87b10324f78-dbus\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-device-dir\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-etc-selinux\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bcf3b15-117f-4aaf-a94a-08997ee640c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237081 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-var-lib-kubelet\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237103 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8457194-8923-4c07-9e06-b06463781252-ovnkube-script-lib\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bcf3b15-117f-4aaf-a94a-08997ee640c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-run-netns\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-run-systemd\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd7qg\" (UniqueName: \"kubernetes.io/projected/e8457194-8923-4c07-9e06-b06463781252-kube-api-access-gd7qg\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237213 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-sysctl-conf\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-multus-conf-dir\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-run-multus-certs\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ae16cac-a8e0-4e04-b938-da9258b3a883-iptables-alerter-script\") pod \"iptables-alerter-hcl5q\" (UID: \"7ae16cac-a8e0-4e04-b938-da9258b3a883\") " pod="openshift-network-operator/iptables-alerter-hcl5q" Sep 19 10:34:32.237633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237377 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-run-multus-certs\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237441 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-systemd-units\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237491 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4345b3c-abd0-4cc2-8b21-a87b10324f78-kubelet-config\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237525 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-lib-modules\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237564 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-sys\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237526 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bcf3b15-117f-4aaf-a94a-08997ee640c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237569 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ae16cac-a8e0-4e04-b938-da9258b3a883-host-slash\") pod \"iptables-alerter-hcl5q\" (UID: \"7ae16cac-a8e0-4e04-b938-da9258b3a883\") " pod="openshift-network-operator/iptables-alerter-hcl5q" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.237443 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237619 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-kubelet-dir\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237529 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-run\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237661 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-var-lib-kubelet\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237693 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-sysctl-d\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237753 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-sysconfig\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237825 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/712bf5a4-5979-4e67-bab1-92d6befbf2e8-cni-binary-copy\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-modprobe-d\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238051 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/712bf5a4-5979-4e67-bab1-92d6befbf2e8-multus-daemon-config\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238100 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-host\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238281 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-cni-netd\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.238325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/747ac66b-6228-471f-9409-77c6246911dd-serviceca\") pod \"node-ca-k2ddk\" (UID: \"747ac66b-6228-471f-9409-77c6246911dd\") " pod="openshift-image-registry/node-ca-k2ddk" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238319 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-system-cni-dir\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238325 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-run-ovn-kubernetes\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.238365 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238365 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-run-openvswitch\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238377 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-device-dir\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238408 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-kubelet\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.238425 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs podName:1728a47c-9260-477e-8b40-42c3cfdee6db nodeName:}" failed. No retries permitted until 2026-09-19 10:34:32.738405414 +0000 UTC m=+4.650910116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs") pod "network-metrics-daemon-9xvrh" (UID: "1728a47c-9260-477e-8b40-42c3cfdee6db") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238429 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e5b260f-bb0a-4096-9a9f-f4a534e77eea-tmp-dir\") pod \"node-resolver-c9c22\" (UID: \"9e5b260f-bb0a-4096-9a9f-f4a534e77eea\") " pod="openshift-dns/node-resolver-c9c22" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238461 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-etc-selinux\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-var-lib-openvswitch\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238480 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-var-lib-cni-bin\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-registration-dir\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238633 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bcf3b15-117f-4aaf-a94a-08997ee640c8-cnibin\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238751 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-etc-kubernetes\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238751 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bcf3b15-117f-4aaf-a94a-08997ee640c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.239166 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.238810 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret podName:f4345b3c-abd0-4cc2-8b21-a87b10324f78 nodeName:}" failed. No retries permitted until 2026-09-19 10:34:32.738800855 +0000 UTC m=+4.651305562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret") pod "global-pull-secret-syncer-d2djw" (UID: "f4345b3c-abd0-4cc2-8b21-a87b10324f78") : object "kube-system"/"original-pull-secret" not registered Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238863 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-run-netns\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35d5cc80-a8a7-4ce0-8195-9464476aae21-socket-dir\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238977 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d60a105e-478f-4a4b-b1b9-daff4de47c93-konnectivity-ca\") pod \"konnectivity-agent-jzzq5\" (UID: \"d60a105e-478f-4a4b-b1b9-daff4de47c93\") " pod="kube-system/konnectivity-agent-jzzq5" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.238986 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bcf3b15-117f-4aaf-a94a-08997ee640c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239004 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.237117 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-run-k8s-cni-cncf-io\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239050 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-cnibin\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239071 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-host-run-netns\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239079 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-multus-cni-dir\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239103 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-run-ovn\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239115 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-hostroot\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8457194-8923-4c07-9e06-b06463781252-env-overrides\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bcf3b15-117f-4aaf-a94a-08997ee640c8-system-cni-dir\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b32d1889-b166-48b3-979f-8a99b875bb78-etc-kubernetes\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239209 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-host-var-lib-cni-multus\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e5b260f-bb0a-4096-9a9f-f4a534e77eea-hosts-file\") pod \"node-resolver-c9c22\" (UID: \"9e5b260f-bb0a-4096-9a9f-f4a534e77eea\") " pod="openshift-dns/node-resolver-c9c22" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-log-socket\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239293 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/712bf5a4-5979-4e67-bab1-92d6befbf2e8-multus-conf-dir\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.240665 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.239298 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8457194-8923-4c07-9e06-b06463781252-run-systemd\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.243272 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.242991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b32d1889-b166-48b3-979f-8a99b875bb78-etc-tuned\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.243272 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.243230 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8457194-8923-4c07-9e06-b06463781252-ovn-node-metrics-cert\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.243272 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.243242 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d60a105e-478f-4a4b-b1b9-daff4de47c93-agent-certs\") pod \"konnectivity-agent-jzzq5\" (UID: \"d60a105e-478f-4a4b-b1b9-daff4de47c93\") " pod="kube-system/konnectivity-agent-jzzq5" Sep 19 10:34:32.243923 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.243886 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b32d1889-b166-48b3-979f-8a99b875bb78-tmp\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.247078 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.247036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjmh8\" (UniqueName: \"kubernetes.io/projected/b32d1889-b166-48b3-979f-8a99b875bb78-kube-api-access-fjmh8\") pod \"tuned-cljtb\" (UID: \"b32d1889-b166-48b3-979f-8a99b875bb78\") " pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.250690 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.250667 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m62j\" (UniqueName: \"kubernetes.io/projected/35d5cc80-a8a7-4ce0-8195-9464476aae21-kube-api-access-6m62j\") pod \"aws-ebs-csi-driver-node-76wtz\" (UID: \"35d5cc80-a8a7-4ce0-8195-9464476aae21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.252214 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.252187 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjw6\" (UniqueName: \"kubernetes.io/projected/1728a47c-9260-477e-8b40-42c3cfdee6db-kube-api-access-drjw6\") pod \"network-metrics-daemon-9xvrh\" (UID: \"1728a47c-9260-477e-8b40-42c3cfdee6db\") " pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:32.252861 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.252842 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 10:34:32.252939 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.252871 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 10:34:32.252939 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.252883 2579 projected.go:196] Error preparing data for projected volume kube-api-access-npj58 for pod openshift-network-diagnostics/network-check-target-ntfqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:34:32.253049 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.252944 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58 podName:65cc3016-6e8a-4a8c-be7b-3634a31508a1 nodeName:}" failed. No retries permitted until 2026-09-19 10:34:32.752927697 +0000 UTC m=+4.665432409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-npj58" (UniqueName: "kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58") pod "network-check-target-ntfqc" (UID: "65cc3016-6e8a-4a8c-be7b-3634a31508a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:34:32.254296 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.253379 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49bgv\" (UniqueName: \"kubernetes.io/projected/9e5b260f-bb0a-4096-9a9f-f4a534e77eea-kube-api-access-49bgv\") pod \"node-resolver-c9c22\" (UID: \"9e5b260f-bb0a-4096-9a9f-f4a534e77eea\") " pod="openshift-dns/node-resolver-c9c22" Sep 19 10:34:32.254296 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.253784 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvwcj\" (UniqueName: \"kubernetes.io/projected/7ae16cac-a8e0-4e04-b938-da9258b3a883-kube-api-access-tvwcj\") pod \"iptables-alerter-hcl5q\" (UID: \"7ae16cac-a8e0-4e04-b938-da9258b3a883\") " pod="openshift-network-operator/iptables-alerter-hcl5q" Sep 19 10:34:32.257740 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.257723 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb8bc\" (UniqueName: \"kubernetes.io/projected/5bcf3b15-117f-4aaf-a94a-08997ee640c8-kube-api-access-tb8bc\") pod \"multus-additional-cni-plugins-2jrfz\" (UID: \"5bcf3b15-117f-4aaf-a94a-08997ee640c8\") " pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.258177 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.258140 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd7qg\" (UniqueName: \"kubernetes.io/projected/e8457194-8923-4c07-9e06-b06463781252-kube-api-access-gd7qg\") pod \"ovnkube-node-jzdrv\" (UID: \"e8457194-8923-4c07-9e06-b06463781252\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.259508 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.259460 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjknj\" (UniqueName: \"kubernetes.io/projected/747ac66b-6228-471f-9409-77c6246911dd-kube-api-access-cjknj\") pod \"node-ca-k2ddk\" (UID: \"747ac66b-6228-471f-9409-77c6246911dd\") " pod="openshift-image-registry/node-ca-k2ddk" Sep 19 10:34:32.259508 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.259481 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhp7f\" (UniqueName: \"kubernetes.io/projected/712bf5a4-5979-4e67-bab1-92d6befbf2e8-kube-api-access-jhp7f\") pod \"multus-dpm4k\" (UID: \"712bf5a4-5979-4e67-bab1-92d6befbf2e8\") " pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.401837 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.401800 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k2ddk" Sep 19 10:34:32.409999 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.409977 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" Sep 19 10:34:32.418824 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.418802 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2jrfz" Sep 19 10:34:32.425556 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.425534 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dpm4k" Sep 19 10:34:32.436528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.436371 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:32.445115 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.445098 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hcl5q" Sep 19 10:34:32.452821 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.452798 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jzzq5" Sep 19 10:34:32.463436 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.463416 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cljtb" Sep 19 10:34:32.469024 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.469007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c9c22" Sep 19 10:34:32.572696 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.572664 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 10:34:32.741584 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.741545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs\") pod \"network-metrics-daemon-9xvrh\" (UID: \"1728a47c-9260-477e-8b40-42c3cfdee6db\") " pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:32.741752 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.741598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:32.741752 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.741716 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:34:32.741880 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.741798 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs podName:1728a47c-9260-477e-8b40-42c3cfdee6db nodeName:}" failed. No retries permitted until 2026-09-19 10:34:33.741777581 +0000 UTC m=+5.654282294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs") pod "network-metrics-daemon-9xvrh" (UID: "1728a47c-9260-477e-8b40-42c3cfdee6db") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:34:32.741944 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.741720 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 10:34:32.742009 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.741998 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret podName:f4345b3c-abd0-4cc2-8b21-a87b10324f78 nodeName:}" failed. No retries permitted until 2026-09-19 10:34:33.741982355 +0000 UTC m=+5.654487066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret") pod "global-pull-secret-syncer-d2djw" (UID: "f4345b3c-abd0-4cc2-8b21-a87b10324f78") : object "kube-system"/"original-pull-secret" not registered Sep 19 10:34:32.842291 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:32.842252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npj58\" (UniqueName: \"kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58\") pod \"network-check-target-ntfqc\" (UID: \"65cc3016-6e8a-4a8c-be7b-3634a31508a1\") " pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:32.842443 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.842428 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 10:34:32.842487 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.842453 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 10:34:32.842487 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.842465 2579 projected.go:196] Error preparing data for projected volume kube-api-access-npj58 for pod openshift-network-diagnostics/network-check-target-ntfqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:34:32.842588 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:32.842529 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58 podName:65cc3016-6e8a-4a8c-be7b-3634a31508a1 nodeName:}" failed. No retries permitted until 2026-09-19 10:34:33.842515959 +0000 UTC m=+5.755020669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-npj58" (UniqueName: "kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58") pod "network-check-target-ntfqc" (UID: "65cc3016-6e8a-4a8c-be7b-3634a31508a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:34:33.163689 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:33.163642 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 10:29:32 +0000 UTC" deadline="2028-06-01 01:51:29.328882452 +0000 UTC" Sep 19 10:34:33.163689 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:33.163676 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14895h16m56.165210165s" Sep 19 10:34:33.190075 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:33.190026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:33.190245 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:33.190164 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:33.392950 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:33.392908 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcf3b15_117f_4aaf_a94a_08997ee640c8.slice/crio-b17f860cc9e9099211b120d08003b977cc2002915459811560fa7d357b035cbe WatchSource:0}: Error finding container b17f860cc9e9099211b120d08003b977cc2002915459811560fa7d357b035cbe: Status 404 returned error can't find the container with id b17f860cc9e9099211b120d08003b977cc2002915459811560fa7d357b035cbe Sep 19 10:34:33.393668 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:33.393638 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8457194_8923_4c07_9e06_b06463781252.slice/crio-99dec86a239625d34e425ac153adf31f481406b3e67c283979db9019c2bd7e69 WatchSource:0}: Error finding container 99dec86a239625d34e425ac153adf31f481406b3e67c283979db9019c2bd7e69: Status 404 returned error can't find the container with id 99dec86a239625d34e425ac153adf31f481406b3e67c283979db9019c2bd7e69 Sep 19 10:34:33.394878 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:33.394824 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32d1889_b166_48b3_979f_8a99b875bb78.slice/crio-2e0009efa04fd4655d176d19776cd5bd8ee700ad7da7cb6a4252379d46429fe8 WatchSource:0}: Error finding container 2e0009efa04fd4655d176d19776cd5bd8ee700ad7da7cb6a4252379d46429fe8: Status 404 returned error can't find the container with id 2e0009efa04fd4655d176d19776cd5bd8ee700ad7da7cb6a4252379d46429fe8 Sep 19 10:34:33.401977 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:33.401954 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5b260f_bb0a_4096_9a9f_f4a534e77eea.slice/crio-2900fbbf732b27783c1fd7c96f7403385b44d90542231be062c5d3bcda072936 WatchSource:0}: Error finding container 2900fbbf732b27783c1fd7c96f7403385b44d90542231be062c5d3bcda072936: Status 404 returned error can't find the container with id 2900fbbf732b27783c1fd7c96f7403385b44d90542231be062c5d3bcda072936 Sep 19 10:34:33.403318 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:34:33.403292 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae16cac_a8e0_4e04_b938_da9258b3a883.slice/crio-8a6930c2997d804e9911bf8b54aab9c33a2390f9fe1fc5262c122305f7384092 WatchSource:0}: Error finding container 8a6930c2997d804e9911bf8b54aab9c33a2390f9fe1fc5262c122305f7384092: Status 404 returned error can't find the container with id 8a6930c2997d804e9911bf8b54aab9c33a2390f9fe1fc5262c122305f7384092 Sep 19 10:34:33.654896 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:33.654875 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 10:34:33.748008 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:33.747939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs\") pod \"network-metrics-daemon-9xvrh\" (UID: \"1728a47c-9260-477e-8b40-42c3cfdee6db\") " pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:33.748008 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:33.747972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:33.748171 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:33.748107 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 10:34:33.748171 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:33.748111 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:34:33.748171 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:33.748159 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret podName:f4345b3c-abd0-4cc2-8b21-a87b10324f78 nodeName:}" failed. No retries permitted until 2026-09-19 10:34:35.748145793 +0000 UTC m=+7.660650490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret") pod "global-pull-secret-syncer-d2djw" (UID: "f4345b3c-abd0-4cc2-8b21-a87b10324f78") : object "kube-system"/"original-pull-secret" not registered Sep 19 10:34:33.748268 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:33.748174 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs podName:1728a47c-9260-477e-8b40-42c3cfdee6db nodeName:}" failed. No retries permitted until 2026-09-19 10:34:35.748167339 +0000 UTC m=+7.660672033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs") pod "network-metrics-daemon-9xvrh" (UID: "1728a47c-9260-477e-8b40-42c3cfdee6db") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:34:33.849231 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:33.849202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npj58\" (UniqueName: \"kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58\") pod \"network-check-target-ntfqc\" (UID: \"65cc3016-6e8a-4a8c-be7b-3634a31508a1\") " pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:33.849358 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:33.849317 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 10:34:33.849358 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:33.849332 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 10:34:33.849358 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:33.849340 2579 projected.go:196] Error preparing data for projected volume kube-api-access-npj58 for pod openshift-network-diagnostics/network-check-target-ntfqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:34:33.849453 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:33.849380 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58 podName:65cc3016-6e8a-4a8c-be7b-3634a31508a1 nodeName:}" failed. No retries permitted until 2026-09-19 10:34:35.849368539 +0000 UTC m=+7.761873233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-npj58" (UniqueName: "kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58") pod "network-check-target-ntfqc" (UID: "65cc3016-6e8a-4a8c-be7b-3634a31508a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:34:34.165763 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.165721 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 10:29:32 +0000 UTC" deadline="2028-03-28 18:31:09.37130423 +0000 UTC" Sep 19 10:34:34.165763 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.165762 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13351h56m35.205546227s" Sep 19 10:34:34.194074 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.192337 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:34.194074 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:34.192454 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:34.194074 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.192834 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:34.194074 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:34.192920 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:34.233137 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.233105 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" event={"ID":"54f93cd73487bad8dda8b78673c5c557","Type":"ContainerStarted","Data":"f4080b74b9e63305959fc1afef9b9e968145bacb60ff4b2d5511d7aaf33a8e52"} Sep 19 10:34:34.246074 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.238882 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jzzq5" event={"ID":"d60a105e-478f-4a4b-b1b9-daff4de47c93","Type":"ContainerStarted","Data":"57523b48ff3898fff696eae052ec61465eaf39a714f223e5934e10025c34dd4d"} Sep 19 10:34:34.246074 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.242547 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" event={"ID":"e8457194-8923-4c07-9e06-b06463781252","Type":"ContainerStarted","Data":"99dec86a239625d34e425ac153adf31f481406b3e67c283979db9019c2bd7e69"} Sep 19 10:34:34.253630 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.253581 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dpm4k" event={"ID":"712bf5a4-5979-4e67-bab1-92d6befbf2e8","Type":"ContainerStarted","Data":"1fce8f415d76f21767a4f28808f061aad130b0f3a40a2ec822dc59f270b733f0"} Sep 19 10:34:34.262872 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.262841 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k2ddk" event={"ID":"747ac66b-6228-471f-9409-77c6246911dd","Type":"ContainerStarted","Data":"250ed85b3204dae04fda8e7cd04db2c4799472405f18515c82d915c3657c3bf1"} Sep 19 10:34:34.290046 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.290008 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" event={"ID":"35d5cc80-a8a7-4ce0-8195-9464476aae21","Type":"ContainerStarted","Data":"18a2fc6276b3f54841510fcdbe9043ccc35cb82d726c96ac635d8f057fe57e46"} Sep 19 10:34:34.302905 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.302873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c9c22" event={"ID":"9e5b260f-bb0a-4096-9a9f-f4a534e77eea","Type":"ContainerStarted","Data":"2900fbbf732b27783c1fd7c96f7403385b44d90542231be062c5d3bcda072936"} Sep 19 10:34:34.313573 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.313546 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hcl5q" event={"ID":"7ae16cac-a8e0-4e04-b938-da9258b3a883","Type":"ContainerStarted","Data":"8a6930c2997d804e9911bf8b54aab9c33a2390f9fe1fc5262c122305f7384092"} Sep 19 10:34:34.321847 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.321819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cljtb" event={"ID":"b32d1889-b166-48b3-979f-8a99b875bb78","Type":"ContainerStarted","Data":"2e0009efa04fd4655d176d19776cd5bd8ee700ad7da7cb6a4252379d46429fe8"} Sep 19 10:34:34.336083 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.333325 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jrfz" event={"ID":"5bcf3b15-117f-4aaf-a94a-08997ee640c8","Type":"ContainerStarted","Data":"b17f860cc9e9099211b120d08003b977cc2002915459811560fa7d357b035cbe"} Sep 19 10:34:34.341755 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.341728 2579 generic.go:356] "Generic (PLEG): container finished" podID="c827f395adacad1f6ed10de19d147344" containerID="315dad7b702f41e1c46f1e3f1c360d9c5e10907c53754599ded22b08d3b0b841" exitCode=0 Sep 19 10:34:34.341861 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.341767 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" event={"ID":"c827f395adacad1f6ed10de19d147344","Type":"ContainerDied","Data":"315dad7b702f41e1c46f1e3f1c360d9c5e10907c53754599ded22b08d3b0b841"} Sep 19 10:34:34.360719 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:34.360668 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" podStartSLOduration=3.360648952 podStartE2EDuration="3.360648952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:34:34.248877115 +0000 UTC m=+6.161381833" watchObservedRunningTime="2026-09-19 10:34:34.360648952 +0000 UTC m=+6.273153669" Sep 19 10:34:35.192279 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:35.192027 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:35.192754 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:35.192417 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:35.361165 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:35.361126 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" event={"ID":"c827f395adacad1f6ed10de19d147344","Type":"ContainerStarted","Data":"343ccdb39e61d958fff7180133c93f69fffbe02136ac62c1a9a99eaf56451a05"} Sep 19 10:34:35.766122 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:35.765258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs\") pod \"network-metrics-daemon-9xvrh\" (UID: \"1728a47c-9260-477e-8b40-42c3cfdee6db\") " pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:35.766122 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:35.765348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:35.766122 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:35.765491 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 10:34:35.766122 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:35.765554 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret podName:f4345b3c-abd0-4cc2-8b21-a87b10324f78 nodeName:}" failed. No retries permitted until 2026-09-19 10:34:39.765536076 +0000 UTC m=+11.678040775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret") pod "global-pull-secret-syncer-d2djw" (UID: "f4345b3c-abd0-4cc2-8b21-a87b10324f78") : object "kube-system"/"original-pull-secret" not registered Sep 19 10:34:35.766122 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:35.766009 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:34:35.766122 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:35.766082 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs podName:1728a47c-9260-477e-8b40-42c3cfdee6db nodeName:}" failed. No retries permitted until 2026-09-19 10:34:39.766051968 +0000 UTC m=+11.678556665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs") pod "network-metrics-daemon-9xvrh" (UID: "1728a47c-9260-477e-8b40-42c3cfdee6db") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:34:35.866750 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:35.866716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npj58\" (UniqueName: \"kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58\") pod \"network-check-target-ntfqc\" (UID: \"65cc3016-6e8a-4a8c-be7b-3634a31508a1\") " pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:35.866924 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:35.866901 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 10:34:35.866995 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:35.866926 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 10:34:35.866995 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:35.866940 2579 projected.go:196] Error preparing data for projected volume kube-api-access-npj58 for pod openshift-network-diagnostics/network-check-target-ntfqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:34:35.867159 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:35.866998 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58 podName:65cc3016-6e8a-4a8c-be7b-3634a31508a1 nodeName:}" failed. No retries permitted until 2026-09-19 10:34:39.866980054 +0000 UTC m=+11.779484766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-npj58" (UniqueName: "kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58") pod "network-check-target-ntfqc" (UID: "65cc3016-6e8a-4a8c-be7b-3634a31508a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:34:36.192909 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:36.191842 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:36.192909 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:36.191966 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:36.192909 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:36.192374 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:36.192909 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:36.192463 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:37.189454 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:37.189421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:37.189663 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:37.189559 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:38.189713 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:38.189653 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:38.190186 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:38.189780 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:38.190186 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:38.190125 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:38.190363 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:38.190228 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:39.189461 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:39.189429 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:39.189625 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:39.189560 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:39.801801 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:39.801769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs\") pod \"network-metrics-daemon-9xvrh\" (UID: \"1728a47c-9260-477e-8b40-42c3cfdee6db\") " pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:39.802294 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:39.801808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:39.802294 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:39.801946 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:34:39.802294 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:39.802019 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs podName:1728a47c-9260-477e-8b40-42c3cfdee6db nodeName:}" failed. No retries permitted until 2026-09-19 10:34:47.801997755 +0000 UTC m=+19.714502464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs") pod "network-metrics-daemon-9xvrh" (UID: "1728a47c-9260-477e-8b40-42c3cfdee6db") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:34:39.802294 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:39.801955 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 10:34:39.802294 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:39.802118 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret podName:f4345b3c-abd0-4cc2-8b21-a87b10324f78 nodeName:}" failed. No retries permitted until 2026-09-19 10:34:47.802100283 +0000 UTC m=+19.714604983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret") pod "global-pull-secret-syncer-d2djw" (UID: "f4345b3c-abd0-4cc2-8b21-a87b10324f78") : object "kube-system"/"original-pull-secret" not registered Sep 19 10:34:39.902900 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:39.902868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npj58\" (UniqueName: \"kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58\") pod \"network-check-target-ntfqc\" (UID: \"65cc3016-6e8a-4a8c-be7b-3634a31508a1\") " pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:39.903082 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:39.903036 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 10:34:39.903082 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:39.903075 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 10:34:39.903186 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:39.903088 2579 projected.go:196] Error preparing data for projected volume kube-api-access-npj58 for pod openshift-network-diagnostics/network-check-target-ntfqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:34:39.903186 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:39.903146 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58 podName:65cc3016-6e8a-4a8c-be7b-3634a31508a1 nodeName:}" failed. No retries permitted until 2026-09-19 10:34:47.903128767 +0000 UTC m=+19.815633478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-npj58" (UniqueName: "kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58") pod "network-check-target-ntfqc" (UID: "65cc3016-6e8a-4a8c-be7b-3634a31508a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:34:40.190599 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:40.190241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:40.190599 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:40.190356 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:40.190599 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:40.190409 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:40.190599 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:40.190524 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:41.189958 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:41.189852 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:41.190346 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:41.189993 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:42.190472 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:42.190437 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:42.190890 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:42.190591 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:42.190890 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:42.190836 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:42.190983 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:42.190936 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:43.189936 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:43.189899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:43.190117 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:43.190023 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:44.190103 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:44.190053 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:44.190103 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:44.190095 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:44.190692 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:44.190193 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:44.190692 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:44.190276 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:45.189795 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:45.189764 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:45.189985 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:45.189900 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:46.189646 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:46.189557 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:46.190116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:46.189565 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:46.190116 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:46.189701 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:46.190116 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:46.189768 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:47.189366 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:47.189335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:47.189536 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:47.189470 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:47.860643 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:47.860613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs\") pod \"network-metrics-daemon-9xvrh\" (UID: \"1728a47c-9260-477e-8b40-42c3cfdee6db\") " pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:47.861124 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:47.860660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:47.861124 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:47.860777 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:34:47.861124 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:47.860856 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs podName:1728a47c-9260-477e-8b40-42c3cfdee6db nodeName:}" failed. No retries permitted until 2026-09-19 10:35:03.860831699 +0000 UTC m=+35.773336400 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs") pod "network-metrics-daemon-9xvrh" (UID: "1728a47c-9260-477e-8b40-42c3cfdee6db") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:34:47.861124 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:47.860786 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 10:34:47.861124 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:47.860950 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret podName:f4345b3c-abd0-4cc2-8b21-a87b10324f78 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:03.860931861 +0000 UTC m=+35.773436574 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret") pod "global-pull-secret-syncer-d2djw" (UID: "f4345b3c-abd0-4cc2-8b21-a87b10324f78") : object "kube-system"/"original-pull-secret" not registered Sep 19 10:34:47.961143 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:47.961112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npj58\" (UniqueName: \"kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58\") pod \"network-check-target-ntfqc\" (UID: \"65cc3016-6e8a-4a8c-be7b-3634a31508a1\") " pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:47.961311 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:47.961287 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 10:34:47.961311 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:47.961307 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 10:34:47.961418 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:47.961315 2579 projected.go:196] Error preparing data for projected volume kube-api-access-npj58 for pod openshift-network-diagnostics/network-check-target-ntfqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:34:47.961418 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:47.961365 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58 podName:65cc3016-6e8a-4a8c-be7b-3634a31508a1 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:03.96134978 +0000 UTC m=+35.873854487 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-npj58" (UniqueName: "kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58") pod "network-check-target-ntfqc" (UID: "65cc3016-6e8a-4a8c-be7b-3634a31508a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:34:48.189954 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:48.189924 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:48.190514 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:48.190476 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:48.190870 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:48.189923 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:48.190870 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:48.190817 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:49.189783 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:49.189752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:49.190192 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:49.189872 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:50.190928 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:50.190889 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:50.191412 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:50.191000 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:50.191412 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:50.191053 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:50.191412 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:50.191172 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:51.190014 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.189990 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:51.190185 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:51.190162 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:51.397956 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.397924 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c9c22" event={"ID":"9e5b260f-bb0a-4096-9a9f-f4a534e77eea","Type":"ContainerStarted","Data":"69f4a556161aaa36425a3a12020232907a2c8afcdaf7807bc2577c017f56f47d"} Sep 19 10:34:51.399564 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.399537 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cljtb" event={"ID":"b32d1889-b166-48b3-979f-8a99b875bb78","Type":"ContainerStarted","Data":"a70e856a40a17708cef25bfddd74cbbe7dc7e7e5148740fbf51747d69ac4b480"} Sep 19 10:34:51.402637 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.402608 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bcf3b15-117f-4aaf-a94a-08997ee640c8" containerID="eece77cd9a3991611eb9114f7ef5577df9793c53750398f229b0bbebf9ed2f73" exitCode=0 Sep 19 10:34:51.402730 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.402701 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jrfz" event={"ID":"5bcf3b15-117f-4aaf-a94a-08997ee640c8","Type":"ContainerDied","Data":"eece77cd9a3991611eb9114f7ef5577df9793c53750398f229b0bbebf9ed2f73"} Sep 19 10:34:51.404955 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.404916 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jzzq5" event={"ID":"d60a105e-478f-4a4b-b1b9-daff4de47c93","Type":"ContainerStarted","Data":"15bba687f6c563e65673729b0184d90f4f7e45bf02ba4380c3e30f124507f18f"} Sep 19 10:34:51.408215 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.408198 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" event={"ID":"e8457194-8923-4c07-9e06-b06463781252","Type":"ContainerStarted","Data":"c41af5e02dcbd68bbc9e741e882e8aa4fb80d0855734c65d6faa7358a6b21bd1"} Sep 19 10:34:51.408276 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.408250 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" event={"ID":"e8457194-8923-4c07-9e06-b06463781252","Type":"ContainerStarted","Data":"92b784fefcc55ec7d8c18097fb37d6be22224728830a359769959462b8ab447a"} Sep 19 10:34:51.408320 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.408280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" event={"ID":"e8457194-8923-4c07-9e06-b06463781252","Type":"ContainerStarted","Data":"ab2f7745dff1c0e5154cb295b0c458bb7320a6141e3eb6d229c8024bd5943d11"} Sep 19 10:34:51.408320 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.408314 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" event={"ID":"e8457194-8923-4c07-9e06-b06463781252","Type":"ContainerStarted","Data":"ee5b8855909218a44e4f8674b7ab5eb737e64af83d3642af4550b645506bb4ca"} Sep 19 10:34:51.408413 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.408359 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" event={"ID":"e8457194-8923-4c07-9e06-b06463781252","Type":"ContainerStarted","Data":"27ab92655b4d61f6ba7e319b53ba6a4cddaff641aca20340753ecc528aacfc36"} Sep 19 10:34:51.408413 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.408390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" event={"ID":"e8457194-8923-4c07-9e06-b06463781252","Type":"ContainerStarted","Data":"ee06d2a86cbee3cda3a2a86c952175568940f289ecb9536b1271025bd588f02b"} Sep 19 10:34:51.409734 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.409716 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dpm4k" event={"ID":"712bf5a4-5979-4e67-bab1-92d6befbf2e8","Type":"ContainerStarted","Data":"05ddcc9e3ecfcbb4937662b42a9c7863889abfccae90233382422a6a682b7f3b"} Sep 19 10:34:51.411174 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.411154 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k2ddk" event={"ID":"747ac66b-6228-471f-9409-77c6246911dd","Type":"ContainerStarted","Data":"8c765cb8d941add8e2974ab1acc34422068235fafc162e5f809e4e3113facb4d"} Sep 19 10:34:51.412540 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.412521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" event={"ID":"35d5cc80-a8a7-4ce0-8195-9464476aae21","Type":"ContainerStarted","Data":"696ed6db3045863c33e99539fd4318fbfcdce2bdd90bce0263b2e07bfba9fcfd"} Sep 19 10:34:51.441451 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.441413 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c9c22" podStartSLOduration=3.979971435 podStartE2EDuration="21.441401636s" totalImagesPullingTime="17.461430201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:34:51.441053311 +0000 UTC m=+23.353558039" watchObservedRunningTime="2026-09-19 10:34:51.441401636 +0000 UTC m=+23.353906351" Sep 19 10:34:51.441536 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.441515 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" podStartSLOduration=20.44150813 podStartE2EDuration="20.44150813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:34:35.378363842 +0000 UTC m=+7.290868558" watchObservedRunningTime="2026-09-19 10:34:51.44150813 +0000 UTC m=+23.354012845" Sep 19 10:34:51.483679 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.483608 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jzzq5" podStartSLOduration=7.885161645 podStartE2EDuration="21.483594496s" totalImagesPullingTime="13.598432851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:34:51.483340386 +0000 UTC m=+23.395845101" watchObservedRunningTime="2026-09-19 10:34:51.483594496 +0000 UTC m=+23.396099212" Sep 19 10:34:51.501766 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.501731 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cljtb" podStartSLOduration=4.033495497 podStartE2EDuration="21.50171931s" totalImagesPullingTime="17.468223813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:34:51.501453778 +0000 UTC m=+23.413958494" watchObservedRunningTime="2026-09-19 10:34:51.50171931 +0000 UTC m=+23.414224026" Sep 19 10:34:51.519716 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.519682 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dpm4k" podStartSLOduration=4.046875237 podStartE2EDuration="21.51967046s" totalImagesPullingTime="17.472795223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:34:51.519318479 +0000 UTC m=+23.431823195" watchObservedRunningTime="2026-09-19 10:34:51.51967046 +0000 UTC m=+23.432175241" Sep 19 10:34:51.540001 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:51.539966 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k2ddk" podStartSLOduration=4.08301462 podStartE2EDuration="21.539956113s" totalImagesPullingTime="17.456941493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:34:51.53911637 +0000 UTC m=+23.451621098" watchObservedRunningTime="2026-09-19 10:34:51.539956113 +0000 UTC m=+23.452460828" Sep 19 10:34:52.189909 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:52.189691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:52.190055 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:52.189733 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:52.190055 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:52.190014 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:52.190191 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:52.190106 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:52.239504 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:52.239420 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 19 10:34:52.416970 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:52.416936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" event={"ID":"35d5cc80-a8a7-4ce0-8195-9464476aae21","Type":"ContainerStarted","Data":"e8b9b8f3e894f871f210294ba9ab1a2d0d1afb6635f1f186ba0d5f772449005a"} Sep 19 10:34:52.418522 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:52.418476 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hcl5q" event={"ID":"7ae16cac-a8e0-4e04-b938-da9258b3a883","Type":"ContainerStarted","Data":"8f06d3ccfb6326db28f371a6d37f6301d1b3da772ad91517df177e9a387cc983"} Sep 19 10:34:52.433221 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:52.433171 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hcl5q" podStartSLOduration=4.997011225 podStartE2EDuration="22.433153919s" totalImagesPullingTime="17.436142694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:34:52.432729136 +0000 UTC m=+24.345233852" watchObservedRunningTime="2026-09-19 10:34:52.433153919 +0000 UTC m=+24.345658636" Sep 19 10:34:53.190140 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:53.190104 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:53.190300 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:53.190221 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:53.196224 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:53.196140 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T10:34:52.239436984Z","UUID":"ad0f6a06-bd7c-4f42-9dde-3f56d880b71f","Handler":null,"Name":"","Endpoint":""} Sep 19 10:34:53.197974 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:53.197865 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 19 10:34:53.197974 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:53.197900 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 19 10:34:53.221752 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:53.221732 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jzzq5" Sep 19 10:34:53.425392 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:53.425364 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" event={"ID":"35d5cc80-a8a7-4ce0-8195-9464476aae21","Type":"ContainerStarted","Data":"f3d900c015097458bb84a2774e4de6021b4411931129d7b23a955dcf06e29f07"} Sep 19 10:34:53.443357 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:53.443277 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76wtz" podStartSLOduration=4.017524007 podStartE2EDuration="23.443257755s" totalImagesPullingTime="19.425733748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:34:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:34:53.44306937 +0000 UTC m=+25.355574077" watchObservedRunningTime="2026-09-19 10:34:53.443257755 +0000 UTC m=+25.355762468" Sep 19 10:34:54.189806 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:54.189777 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:54.189806 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:54.189796 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:54.190047 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:54.189895 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:54.190047 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:54.190028 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:54.431226 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:54.431190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" event={"ID":"e8457194-8923-4c07-9e06-b06463781252","Type":"ContainerStarted","Data":"b45e6ab2f42930f53f41b6572355a825b38394a31456a66dcd0516f1d3b5ac38"} Sep 19 10:34:55.190130 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:55.190100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:55.190335 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:55.190221 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:55.223617 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:55.223584 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jzzq5" Sep 19 10:34:55.224193 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:55.224176 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jzzq5" Sep 19 10:34:55.434116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:55.434089 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jzzq5" Sep 19 10:34:56.189868 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:56.189708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:56.189969 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:56.189769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:56.190024 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:56.189934 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:56.190107 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:56.190047 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:56.438661 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:56.438639 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" event={"ID":"e8457194-8923-4c07-9e06-b06463781252","Type":"ContainerStarted","Data":"85ff96fc3c66cab76ca71706f8784a3b50ac4661a54150be73e17eca93c288db"} Sep 19 10:34:56.439267 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:56.438907 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:56.439267 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:56.438970 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:56.440689 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:56.440668 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bcf3b15-117f-4aaf-a94a-08997ee640c8" containerID="1e10aaa90d1cf813f021d971b5f71f3836c1e8a79cec9b31e01a8bf076f8e66f" exitCode=0 Sep 19 10:34:56.440793 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:56.440748 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jrfz" event={"ID":"5bcf3b15-117f-4aaf-a94a-08997ee640c8","Type":"ContainerDied","Data":"1e10aaa90d1cf813f021d971b5f71f3836c1e8a79cec9b31e01a8bf076f8e66f"} Sep 19 10:34:56.454253 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:56.454235 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:56.454341 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:56.454319 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:56.466087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:56.466026 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" podStartSLOduration=8.959818288 podStartE2EDuration="26.466013116s" totalImagesPullingTime="17.506194828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:34:56.464648923 +0000 UTC m=+28.377153651" watchObservedRunningTime="2026-09-19 10:34:56.466013116 +0000 UTC m=+28.378517831" Sep 19 10:34:57.189443 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:57.189414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:57.189587 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:57.189526 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:57.443535 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:57.443472 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:34:57.714809 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:57.714739 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d2djw"] Sep 19 10:34:57.714939 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:57.714845 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:57.714992 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:57.714937 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:34:57.718286 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:57.718261 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ntfqc"] Sep 19 10:34:57.718404 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:57.718352 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:57.718462 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:57.718436 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:57.718993 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:57.718972 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9xvrh"] Sep 19 10:34:57.719094 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:57.719080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:57.719177 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:57.719162 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:58.446494 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:58.446457 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bcf3b15-117f-4aaf-a94a-08997ee640c8" containerID="bc6bbf286dc7e01b8cc11be3abaa8efede4feedb5b7606d2b1d7a762a3fc98b2" exitCode=0 Sep 19 10:34:58.446845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:58.446533 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jrfz" event={"ID":"5bcf3b15-117f-4aaf-a94a-08997ee640c8","Type":"ContainerDied","Data":"bc6bbf286dc7e01b8cc11be3abaa8efede4feedb5b7606d2b1d7a762a3fc98b2"} Sep 19 10:34:59.189563 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:59.189530 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:34:59.189733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:59.189574 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:34:59.189733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:34:59.189635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:34:59.189733 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:59.189650 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:34:59.189733 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:59.189714 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:34:59.189935 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:34:59.189796 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:35:00.453493 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:00.453463 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bcf3b15-117f-4aaf-a94a-08997ee640c8" containerID="375b81ec5e3c75a6f961c7f530bd55fcc58f1961b7199f25bfbcf7424584572d" exitCode=0 Sep 19 10:35:00.453853 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:00.453525 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jrfz" event={"ID":"5bcf3b15-117f-4aaf-a94a-08997ee640c8","Type":"ContainerDied","Data":"375b81ec5e3c75a6f961c7f530bd55fcc58f1961b7199f25bfbcf7424584572d"} Sep 19 10:35:01.189846 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:01.189815 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:35:01.190003 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:01.189853 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:35:01.190003 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:01.189943 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:35:01.190142 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:01.189996 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:35:01.190142 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:01.190070 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:35:01.190241 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:01.190189 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:35:03.190110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.190073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:35:03.190662 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.190117 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:35:03.190662 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.190118 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:35:03.190662 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.190182 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2djw" podUID="f4345b3c-abd0-4cc2-8b21-a87b10324f78" Sep 19 10:35:03.190662 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.190274 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xvrh" podUID="1728a47c-9260-477e-8b40-42c3cfdee6db" Sep 19 10:35:03.190662 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.190340 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntfqc" podUID="65cc3016-6e8a-4a8c-be7b-3634a31508a1" Sep 19 10:35:03.654770 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.654746 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeReady" Sep 19 10:35:03.654934 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.654913 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 19 10:35:03.691609 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.691582 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7774cfc878-x5gcd"] Sep 19 10:35:03.722615 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.722546 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7774cfc878-x5gcd"] Sep 19 10:35:03.722615 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.722586 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gvn74"] Sep 19 10:35:03.722863 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.722846 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.725588 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.725515 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 19 10:35:03.725588 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.725537 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 19 10:35:03.726500 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.725914 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 19 10:35:03.726500 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.726087 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-59nw8\"" Sep 19 10:35:03.731444 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.731422 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 19 10:35:03.737742 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.737720 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4cgqn"] Sep 19 10:35:03.737837 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.737821 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:03.740476 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.740454 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 19 10:35:03.740476 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.740467 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 19 10:35:03.740598 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.740452 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-snddw\"" Sep 19 10:35:03.754048 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.754025 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4cgqn"] Sep 19 10:35:03.754162 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.754074 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gvn74"] Sep 19 10:35:03.754162 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.754112 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4cgqn" Sep 19 10:35:03.756836 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.756814 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 19 10:35:03.756935 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.756868 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 19 10:35:03.757102 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.757087 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cttpt\"" Sep 19 10:35:03.758499 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.758479 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 19 10:35:03.890600 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.890566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13a1b6fa-ab80-4027-981f-5daf44a44b61-trusted-ca\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.890600 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.890607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs\") pod \"network-metrics-daemon-9xvrh\" (UID: \"1728a47c-9260-477e-8b40-42c3cfdee6db\") " pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:35:03.890831 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.890630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.890831 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.890663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:35:03.890831 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.890749 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 10:35:03.890831 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.890775 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a046e7f-51bb-42f3-be5a-e3f15a80afce-tmp-dir\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:03.890831 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.890802 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret podName:f4345b3c-abd0-4cc2-8b21-a87b10324f78 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:35.890789862 +0000 UTC m=+67.803294556 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret") pod "global-pull-secret-syncer-d2djw" (UID: "f4345b3c-abd0-4cc2-8b21-a87b10324f78") : object "kube-system"/"original-pull-secret" not registered Sep 19 10:35:03.890831 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.890749 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:35:03.891075 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.890865 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs podName:1728a47c-9260-477e-8b40-42c3cfdee6db nodeName:}" failed. No retries permitted until 2026-09-19 10:35:35.890850923 +0000 UTC m=+67.803355627 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs") pod "network-metrics-daemon-9xvrh" (UID: "1728a47c-9260-477e-8b40-42c3cfdee6db") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 10:35:03.891075 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.890893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfj8n\" (UniqueName: \"kubernetes.io/projected/6a046e7f-51bb-42f3-be5a-e3f15a80afce-kube-api-access-lfj8n\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:03.891075 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.890918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr6lk\" (UniqueName: \"kubernetes.io/projected/c425dda8-86ce-4a68-947b-e32ded00bfba-kube-api-access-rr6lk\") pod \"ingress-canary-4cgqn\" (UID: \"c425dda8-86ce-4a68-947b-e32ded00bfba\") " pod="openshift-ingress-canary/ingress-canary-4cgqn" Sep 19 10:35:03.891075 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.890945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a046e7f-51bb-42f3-be5a-e3f15a80afce-config-volume\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:03.891075 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.890973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-bound-sa-token\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.891075 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.891046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13a1b6fa-ab80-4027-981f-5daf44a44b61-image-registry-private-configuration\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.891347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.891114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlp2l\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-kube-api-access-tlp2l\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.891347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.891141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:03.891347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.891162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c425dda8-86ce-4a68-947b-e32ded00bfba-cert\") pod \"ingress-canary-4cgqn\" (UID: \"c425dda8-86ce-4a68-947b-e32ded00bfba\") " pod="openshift-ingress-canary/ingress-canary-4cgqn" Sep 19 10:35:03.891347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.891213 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-certificates\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.891347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.891264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13a1b6fa-ab80-4027-981f-5daf44a44b61-installation-pull-secrets\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.891347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.891299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13a1b6fa-ab80-4027-981f-5daf44a44b61-ca-trust-extracted\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.991839 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.991758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13a1b6fa-ab80-4027-981f-5daf44a44b61-installation-pull-secrets\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.991839 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.991795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13a1b6fa-ab80-4027-981f-5daf44a44b61-ca-trust-extracted\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.991839 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.991818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13a1b6fa-ab80-4027-981f-5daf44a44b61-trusted-ca\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.992125 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.991929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.992125 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.991987 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a046e7f-51bb-42f3-be5a-e3f15a80afce-tmp-dir\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:03.992125 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.992017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfj8n\" (UniqueName: \"kubernetes.io/projected/6a046e7f-51bb-42f3-be5a-e3f15a80afce-kube-api-access-lfj8n\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:03.992125 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.992042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr6lk\" (UniqueName: \"kubernetes.io/projected/c425dda8-86ce-4a68-947b-e32ded00bfba-kube-api-access-rr6lk\") pod \"ingress-canary-4cgqn\" (UID: \"c425dda8-86ce-4a68-947b-e32ded00bfba\") " pod="openshift-ingress-canary/ingress-canary-4cgqn" Sep 19 10:35:03.992125 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.992094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a046e7f-51bb-42f3-be5a-e3f15a80afce-config-volume\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:03.992125 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.992099 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 19 10:35:03.992125 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.992116 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7774cfc878-x5gcd: secret "image-registry-tls" not found Sep 19 10:35:03.992125 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.992122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-bound-sa-token\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.992528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.992151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npj58\" (UniqueName: \"kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58\") pod \"network-check-target-ntfqc\" (UID: \"65cc3016-6e8a-4a8c-be7b-3634a31508a1\") " pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:35:03.992528 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.992171 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls podName:13a1b6fa-ab80-4027-981f-5daf44a44b61 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:04.492152783 +0000 UTC m=+36.404657491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls") pod "image-registry-7774cfc878-x5gcd" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61") : secret "image-registry-tls" not found Sep 19 10:35:03.992528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.992207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13a1b6fa-ab80-4027-981f-5daf44a44b61-image-registry-private-configuration\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.992528 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.992237 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 10:35:03.992528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.992243 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13a1b6fa-ab80-4027-981f-5daf44a44b61-ca-trust-extracted\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.992528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.992248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlp2l\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-kube-api-access-tlp2l\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.992528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.992306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:03.992528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.992337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c425dda8-86ce-4a68-947b-e32ded00bfba-cert\") pod \"ingress-canary-4cgqn\" (UID: \"c425dda8-86ce-4a68-947b-e32ded00bfba\") " pod="openshift-ingress-canary/ingress-canary-4cgqn" Sep 19 10:35:03.992528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.992365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-certificates\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.992950 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.992253 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 10:35:03.992950 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.992834 2579 projected.go:196] Error preparing data for projected volume kube-api-access-npj58 for pod openshift-network-diagnostics/network-check-target-ntfqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:35:03.992950 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.992879 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58 podName:65cc3016-6e8a-4a8c-be7b-3634a31508a1 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:35.992863942 +0000 UTC m=+67.905368653 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-npj58" (UniqueName: "kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58") pod "network-check-target-ntfqc" (UID: "65cc3016-6e8a-4a8c-be7b-3634a31508a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 10:35:03.992950 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.992812 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 10:35:03.992950 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:03.992920 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls podName:6a046e7f-51bb-42f3-be5a-e3f15a80afce nodeName:}" failed. No retries permitted until 2026-09-19 10:35:04.492909671 +0000 UTC m=+36.405414379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls") pod "dns-default-gvn74" (UID: "6a046e7f-51bb-42f3-be5a-e3f15a80afce") : secret "dns-default-metrics-tls" not found Sep 19 10:35:03.992950 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.992924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13a1b6fa-ab80-4027-981f-5daf44a44b61-trusted-ca\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.993308 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.993086 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-certificates\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.993308 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.993180 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a046e7f-51bb-42f3-be5a-e3f15a80afce-tmp-dir\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:03.993405 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.993345 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a046e7f-51bb-42f3-be5a-e3f15a80afce-config-volume\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:03.996989 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.996969 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13a1b6fa-ab80-4027-981f-5daf44a44b61-installation-pull-secrets\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.997132 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.996988 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13a1b6fa-ab80-4027-981f-5daf44a44b61-image-registry-private-configuration\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:03.997132 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:03.997044 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c425dda8-86ce-4a68-947b-e32ded00bfba-cert\") pod \"ingress-canary-4cgqn\" (UID: \"c425dda8-86ce-4a68-947b-e32ded00bfba\") " pod="openshift-ingress-canary/ingress-canary-4cgqn" Sep 19 10:35:04.001819 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:04.001795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfj8n\" (UniqueName: \"kubernetes.io/projected/6a046e7f-51bb-42f3-be5a-e3f15a80afce-kube-api-access-lfj8n\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:04.001942 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:04.001850 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-bound-sa-token\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:04.001942 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:04.001884 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlp2l\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-kube-api-access-tlp2l\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:04.002229 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:04.002210 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr6lk\" (UniqueName: \"kubernetes.io/projected/c425dda8-86ce-4a68-947b-e32ded00bfba-kube-api-access-rr6lk\") pod \"ingress-canary-4cgqn\" (UID: \"c425dda8-86ce-4a68-947b-e32ded00bfba\") " pod="openshift-ingress-canary/ingress-canary-4cgqn" Sep 19 10:35:04.064583 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:04.064559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4cgqn" Sep 19 10:35:04.496047 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:04.496012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:04.496561 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:04.496102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:04.496561 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:04.496185 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 10:35:04.496561 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:04.496210 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 19 10:35:04.496561 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:04.496225 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7774cfc878-x5gcd: secret "image-registry-tls" not found Sep 19 10:35:04.496561 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:04.496276 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls podName:6a046e7f-51bb-42f3-be5a-e3f15a80afce nodeName:}" failed. No retries permitted until 2026-09-19 10:35:05.496256235 +0000 UTC m=+37.408760931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls") pod "dns-default-gvn74" (UID: "6a046e7f-51bb-42f3-be5a-e3f15a80afce") : secret "dns-default-metrics-tls" not found Sep 19 10:35:04.496561 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:04.496296 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls podName:13a1b6fa-ab80-4027-981f-5daf44a44b61 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:05.496286365 +0000 UTC m=+37.408791059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls") pod "image-registry-7774cfc878-x5gcd" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61") : secret "image-registry-tls" not found Sep 19 10:35:05.190191 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:05.190004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:35:05.190327 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:05.190004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:35:05.190377 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:05.190004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:35:05.194171 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:05.194150 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d9kv4\"" Sep 19 10:35:05.194279 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:05.194185 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 19 10:35:05.194279 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:05.194151 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8jxmp\"" Sep 19 10:35:05.194279 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:05.194150 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 19 10:35:05.194279 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:05.194228 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 19 10:35:05.194279 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:05.194255 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 19 10:35:05.303854 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:05.303801 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4cgqn"] Sep 19 10:35:05.342949 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:05.342910 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc425dda8_86ce_4a68_947b_e32ded00bfba.slice/crio-53728943e4260c9a3de961ca29b1dab26ddc36d4c2ee0549b7fbb87833add11b WatchSource:0}: Error finding container 53728943e4260c9a3de961ca29b1dab26ddc36d4c2ee0549b7fbb87833add11b: Status 404 returned error can't find the container with id 53728943e4260c9a3de961ca29b1dab26ddc36d4c2ee0549b7fbb87833add11b Sep 19 10:35:05.465490 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:05.465454 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4cgqn" event={"ID":"c425dda8-86ce-4a68-947b-e32ded00bfba","Type":"ContainerStarted","Data":"53728943e4260c9a3de961ca29b1dab26ddc36d4c2ee0549b7fbb87833add11b"} Sep 19 10:35:05.503862 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:05.503839 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:05.504192 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:05.503895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:05.504192 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:05.503977 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 10:35:05.504192 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:05.503986 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 19 10:35:05.504192 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:05.504003 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7774cfc878-x5gcd: secret "image-registry-tls" not found Sep 19 10:35:05.504192 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:05.504024 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls podName:6a046e7f-51bb-42f3-be5a-e3f15a80afce nodeName:}" failed. No retries permitted until 2026-09-19 10:35:07.504011757 +0000 UTC m=+39.416516463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls") pod "dns-default-gvn74" (UID: "6a046e7f-51bb-42f3-be5a-e3f15a80afce") : secret "dns-default-metrics-tls" not found Sep 19 10:35:05.504192 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:05.504046 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls podName:13a1b6fa-ab80-4027-981f-5daf44a44b61 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:07.504029039 +0000 UTC m=+39.416533735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls") pod "image-registry-7774cfc878-x5gcd" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61") : secret "image-registry-tls" not found Sep 19 10:35:06.470987 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:06.470954 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bcf3b15-117f-4aaf-a94a-08997ee640c8" containerID="2379a3f10f4e23fd171529ecea0b605906711822b3116fe71114f01176ff35b7" exitCode=0 Sep 19 10:35:06.471154 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:06.471018 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jrfz" event={"ID":"5bcf3b15-117f-4aaf-a94a-08997ee640c8","Type":"ContainerDied","Data":"2379a3f10f4e23fd171529ecea0b605906711822b3116fe71114f01176ff35b7"} Sep 19 10:35:07.477866 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:07.477776 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bcf3b15-117f-4aaf-a94a-08997ee640c8" containerID="cb02800ecb2b5180202b0c6d50ab90dbaabd31283ff98529bc061f49ff6d70a4" exitCode=0 Sep 19 10:35:07.477866 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:07.477845 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jrfz" event={"ID":"5bcf3b15-117f-4aaf-a94a-08997ee640c8","Type":"ContainerDied","Data":"cb02800ecb2b5180202b0c6d50ab90dbaabd31283ff98529bc061f49ff6d70a4"} Sep 19 10:35:07.518942 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:07.518908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:07.519135 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:07.518998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:07.519135 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:07.519107 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 19 10:35:07.519135 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:07.519126 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7774cfc878-x5gcd: secret "image-registry-tls" not found Sep 19 10:35:07.519135 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:07.519129 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 10:35:07.519352 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:07.519183 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls podName:13a1b6fa-ab80-4027-981f-5daf44a44b61 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:11.519164394 +0000 UTC m=+43.431669109 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls") pod "image-registry-7774cfc878-x5gcd" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61") : secret "image-registry-tls" not found Sep 19 10:35:07.519352 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:07.519202 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls podName:6a046e7f-51bb-42f3-be5a-e3f15a80afce nodeName:}" failed. No retries permitted until 2026-09-19 10:35:11.519192572 +0000 UTC m=+43.431697274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls") pod "dns-default-gvn74" (UID: "6a046e7f-51bb-42f3-be5a-e3f15a80afce") : secret "dns-default-metrics-tls" not found Sep 19 10:35:08.076369 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.076347 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4md95"] Sep 19 10:35:08.091593 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.091566 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-xnm4h"] Sep 19 10:35:08.091710 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.091610 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4md95" Sep 19 10:35:08.096272 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.096250 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 19 10:35:08.096390 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.096296 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vmcct\"" Sep 19 10:35:08.096569 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.096549 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 19 10:35:08.114568 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.114547 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62"] Sep 19 10:35:08.114675 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.114660 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.117698 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.117683 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8dxsq\"" Sep 19 10:35:08.117964 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.117949 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 19 10:35:08.118102 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.118082 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 19 10:35:08.118205 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.118087 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 19 10:35:08.118255 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.118225 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 19 10:35:08.123894 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.123877 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 19 10:35:08.138856 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.138835 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8"] Sep 19 10:35:08.138956 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.138942 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" Sep 19 10:35:08.141935 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.141918 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 19 10:35:08.142077 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.142041 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5c5zs\"" Sep 19 10:35:08.142188 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.142172 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 19 10:35:08.142243 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.142237 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 19 10:35:08.142453 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.142441 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 19 10:35:08.159069 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.159040 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4md95"] Sep 19 10:35:08.159175 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.159082 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-xnm4h"] Sep 19 10:35:08.159175 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.159097 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62"] Sep 19 10:35:08.159175 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.159110 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8"] Sep 19 10:35:08.159175 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.159165 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" Sep 19 10:35:08.161714 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.161694 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 19 10:35:08.161824 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.161725 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-g6qhq\"" Sep 19 10:35:08.161824 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.161743 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 19 10:35:08.161824 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.161700 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 19 10:35:08.162169 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.162151 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 19 10:35:08.166691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.166672 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8"] Sep 19 10:35:08.183431 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.183410 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-wxvs7"] Sep 19 10:35:08.183569 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.183552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:08.185905 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.185884 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 19 10:35:08.186187 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.186168 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 19 10:35:08.186284 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.186265 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-54v4j\"" Sep 19 10:35:08.203902 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.203882 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm"] Sep 19 10:35:08.204011 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.203997 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-wxvs7" Sep 19 10:35:08.208296 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.207238 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-97ztb\"" Sep 19 10:35:08.224116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.224011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lssn\" (UniqueName: \"kubernetes.io/projected/697fd709-5743-4f20-aa9b-8cc6285edeaa-kube-api-access-2lssn\") pod \"volume-data-source-validator-5c4db8bdf9-4md95\" (UID: \"697fd709-5743-4f20-aa9b-8cc6285edeaa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4md95" Sep 19 10:35:08.224116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.224048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61826a91-39c7-4d1b-a0f9-c1de1de49c0e-serving-cert\") pod \"console-operator-fd4bb46c-xnm4h\" (UID: \"61826a91-39c7-4d1b-a0f9-c1de1de49c0e\") " pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.224116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.224082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61826a91-39c7-4d1b-a0f9-c1de1de49c0e-config\") pod \"console-operator-fd4bb46c-xnm4h\" (UID: \"61826a91-39c7-4d1b-a0f9-c1de1de49c0e\") " pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.224342 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.224120 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61826a91-39c7-4d1b-a0f9-c1de1de49c0e-trusted-ca\") pod \"console-operator-fd4bb46c-xnm4h\" (UID: \"61826a91-39c7-4d1b-a0f9-c1de1de49c0e\") " pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.224342 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.224137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p868l\" (UniqueName: \"kubernetes.io/projected/61826a91-39c7-4d1b-a0f9-c1de1de49c0e-kube-api-access-p868l\") pod \"console-operator-fd4bb46c-xnm4h\" (UID: \"61826a91-39c7-4d1b-a0f9-c1de1de49c0e\") " pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.228165 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.228146 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-9998785cb-l2lkw"] Sep 19 10:35:08.228459 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.228440 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:08.231255 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.231226 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 19 10:35:08.231439 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.231322 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 19 10:35:08.231439 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.231344 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 19 10:35:08.231539 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.231462 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qkdwx\"" Sep 19 10:35:08.233254 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.233235 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 19 10:35:08.250697 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.250676 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.252996 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.252971 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 19 10:35:08.253119 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.253007 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 19 10:35:08.253119 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.253107 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 19 10:35:08.253509 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.253493 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5b9hv\"" Sep 19 10:35:08.253509 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.253503 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 19 10:35:08.253685 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.253593 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 19 10:35:08.253685 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.253656 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 19 10:35:08.254678 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.254662 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8"] Sep 19 10:35:08.254735 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.254699 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm"] Sep 19 10:35:08.254735 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.254714 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9998785cb-l2lkw"] Sep 19 10:35:08.254735 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.254726 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-wxvs7"] Sep 19 10:35:08.271448 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.271427 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-t5cft"] Sep 19 10:35:08.290304 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.290281 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9"] Sep 19 10:35:08.290425 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.290407 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.292821 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.292801 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 19 10:35:08.292924 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.292894 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 19 10:35:08.293115 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.293092 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 19 10:35:08.293258 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.293178 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mnk9z\"" Sep 19 10:35:08.293258 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.293244 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 19 10:35:08.297714 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.297698 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 19 10:35:08.308633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.308610 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" Sep 19 10:35:08.308714 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.308632 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-t5cft"] Sep 19 10:35:08.308714 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.308672 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9"] Sep 19 10:35:08.311054 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.311038 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 19 10:35:08.311159 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.311088 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 19 10:35:08.311159 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.311104 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 19 10:35:08.311159 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.311131 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c6pdj\"" Sep 19 10:35:08.325406 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5tsz\" (UniqueName: \"kubernetes.io/projected/a1e60166-09e7-42c8-aaea-c82942348bf7-kube-api-access-m5tsz\") pod \"service-ca-operator-57cdc4b768-nn5f8\" (UID: \"a1e60166-09e7-42c8-aaea-c82942348bf7\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" Sep 19 10:35:08.325482 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bdae529f-3e44-4813-906d-fafa5aa85f27-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-2jc62\" (UID: \"bdae529f-3e44-4813-906d-fafa5aa85f27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" Sep 19 10:35:08.325482 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1e60166-09e7-42c8-aaea-c82942348bf7-config\") pod \"service-ca-operator-57cdc4b768-nn5f8\" (UID: \"a1e60166-09e7-42c8-aaea-c82942348bf7\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" Sep 19 10:35:08.325482 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325451 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jsd8\" (UID: \"e1e97edd-62b0-443a-b1cc-d7b796704644\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:08.325649 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61826a91-39c7-4d1b-a0f9-c1de1de49c0e-trusted-ca\") pod \"console-operator-fd4bb46c-xnm4h\" (UID: \"61826a91-39c7-4d1b-a0f9-c1de1de49c0e\") " pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.325649 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bdae529f-3e44-4813-906d-fafa5aa85f27-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-2jc62\" (UID: \"bdae529f-3e44-4813-906d-fafa5aa85f27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" Sep 19 10:35:08.325649 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd29j\" (UniqueName: \"kubernetes.io/projected/c77ef5c2-0b57-4986-b196-8f007104d157-kube-api-access-kd29j\") pod \"network-check-source-7cbc45fddc-wxvs7\" (UID: \"c77ef5c2-0b57-4986-b196-8f007104d157\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-wxvs7" Sep 19 10:35:08.325649 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e1e97edd-62b0-443a-b1cc-d7b796704644-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7jsd8\" (UID: \"e1e97edd-62b0-443a-b1cc-d7b796704644\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:08.325649 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lssn\" (UniqueName: \"kubernetes.io/projected/697fd709-5743-4f20-aa9b-8cc6285edeaa-kube-api-access-2lssn\") pod \"volume-data-source-validator-5c4db8bdf9-4md95\" (UID: \"697fd709-5743-4f20-aa9b-8cc6285edeaa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4md95" Sep 19 10:35:08.325649 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61826a91-39c7-4d1b-a0f9-c1de1de49c0e-serving-cert\") pod \"console-operator-fd4bb46c-xnm4h\" (UID: \"61826a91-39c7-4d1b-a0f9-c1de1de49c0e\") " pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.325937 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/245f29a4-837b-4197-9eb7-52f324d83901-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:08.325937 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:08.325937 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61826a91-39c7-4d1b-a0f9-c1de1de49c0e-config\") pod \"console-operator-fd4bb46c-xnm4h\" (UID: \"61826a91-39c7-4d1b-a0f9-c1de1de49c0e\") " pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.325937 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggkt9\" (UniqueName: \"kubernetes.io/projected/bdae529f-3e44-4813-906d-fafa5aa85f27-kube-api-access-ggkt9\") pod \"kube-storage-version-migrator-operator-55889f7b47-2jc62\" (UID: \"bdae529f-3e44-4813-906d-fafa5aa85f27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" Sep 19 10:35:08.325937 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hphhx\" (UniqueName: \"kubernetes.io/projected/245f29a4-837b-4197-9eb7-52f324d83901-kube-api-access-hphhx\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:08.325937 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1e60166-09e7-42c8-aaea-c82942348bf7-serving-cert\") pod \"service-ca-operator-57cdc4b768-nn5f8\" (UID: \"a1e60166-09e7-42c8-aaea-c82942348bf7\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" Sep 19 10:35:08.326261 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.325959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p868l\" (UniqueName: \"kubernetes.io/projected/61826a91-39c7-4d1b-a0f9-c1de1de49c0e-kube-api-access-p868l\") pod \"console-operator-fd4bb46c-xnm4h\" (UID: \"61826a91-39c7-4d1b-a0f9-c1de1de49c0e\") " pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.326401 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.326381 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61826a91-39c7-4d1b-a0f9-c1de1de49c0e-config\") pod \"console-operator-fd4bb46c-xnm4h\" (UID: \"61826a91-39c7-4d1b-a0f9-c1de1de49c0e\") " pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.326456 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.326386 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61826a91-39c7-4d1b-a0f9-c1de1de49c0e-trusted-ca\") pod \"console-operator-fd4bb46c-xnm4h\" (UID: \"61826a91-39c7-4d1b-a0f9-c1de1de49c0e\") " pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.329203 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.329185 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61826a91-39c7-4d1b-a0f9-c1de1de49c0e-serving-cert\") pod \"console-operator-fd4bb46c-xnm4h\" (UID: \"61826a91-39c7-4d1b-a0f9-c1de1de49c0e\") " pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.335863 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.335838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lssn\" (UniqueName: \"kubernetes.io/projected/697fd709-5743-4f20-aa9b-8cc6285edeaa-kube-api-access-2lssn\") pod \"volume-data-source-validator-5c4db8bdf9-4md95\" (UID: \"697fd709-5743-4f20-aa9b-8cc6285edeaa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4md95" Sep 19 10:35:08.337101 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.337085 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p868l\" (UniqueName: \"kubernetes.io/projected/61826a91-39c7-4d1b-a0f9-c1de1de49c0e-kube-api-access-p868l\") pod \"console-operator-fd4bb46c-xnm4h\" (UID: \"61826a91-39c7-4d1b-a0f9-c1de1de49c0e\") " pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.402701 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.402679 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4md95" Sep 19 10:35:08.423247 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.423229 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:08.426479 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.426461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5tsz\" (UniqueName: \"kubernetes.io/projected/a1e60166-09e7-42c8-aaea-c82942348bf7-kube-api-access-m5tsz\") pod \"service-ca-operator-57cdc4b768-nn5f8\" (UID: \"a1e60166-09e7-42c8-aaea-c82942348bf7\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" Sep 19 10:35:08.426523 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.426488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efe9b96b-1362-4fc4-8701-36225c008ec9-tmp\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.426523 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.426505 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtdw8\" (UniqueName: \"kubernetes.io/projected/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-kube-api-access-rtdw8\") pod \"cluster-samples-operator-685d7d68b5-d49c9\" (UID: \"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" Sep 19 10:35:08.426600 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.426526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bdae529f-3e44-4813-906d-fafa5aa85f27-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-2jc62\" (UID: \"bdae529f-3e44-4813-906d-fafa5aa85f27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" Sep 19 10:35:08.426600 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.426546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1e60166-09e7-42c8-aaea-c82942348bf7-config\") pod \"service-ca-operator-57cdc4b768-nn5f8\" (UID: \"a1e60166-09e7-42c8-aaea-c82942348bf7\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" Sep 19 10:35:08.426600 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.426587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jsd8\" (UID: \"e1e97edd-62b0-443a-b1cc-d7b796704644\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:08.426724 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.426621 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efe9b96b-1362-4fc4-8701-36225c008ec9-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.426724 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.426647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-stats-auth\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.426724 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.426689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bdae529f-3e44-4813-906d-fafa5aa85f27-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-2jc62\" (UID: \"bdae529f-3e44-4813-906d-fafa5aa85f27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" Sep 19 10:35:08.426724 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.426715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.426724 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:08.426718 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 19 10:35:08.426929 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.426766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd29j\" (UniqueName: \"kubernetes.io/projected/c77ef5c2-0b57-4986-b196-8f007104d157-kube-api-access-kd29j\") pod \"network-check-source-7cbc45fddc-wxvs7\" (UID: \"c77ef5c2-0b57-4986-b196-8f007104d157\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-wxvs7" Sep 19 10:35:08.426929 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:08.426775 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert podName:e1e97edd-62b0-443a-b1cc-d7b796704644 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:08.92676156 +0000 UTC m=+40.839266267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7jsd8" (UID: "e1e97edd-62b0-443a-b1cc-d7b796704644") : secret "networking-console-plugin-cert" not found Sep 19 10:35:08.426929 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.426892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d49c9\" (UID: \"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" Sep 19 10:35:08.427086 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e1e97edd-62b0-443a-b1cc-d7b796704644-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7jsd8\" (UID: \"e1e97edd-62b0-443a-b1cc-d7b796704644\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:08.427145 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427097 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-default-certificate\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.427145 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5b9q\" (UniqueName: \"kubernetes.io/projected/6b92f4b1-f19a-42ac-b87b-835f23d214cd-kube-api-access-m5b9q\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.427251 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427143 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bdae529f-3e44-4813-906d-fafa5aa85f27-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-2jc62\" (UID: \"bdae529f-3e44-4813-906d-fafa5aa85f27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" Sep 19 10:35:08.427251 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427158 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/245f29a4-837b-4197-9eb7-52f324d83901-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:08.427251 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427161 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1e60166-09e7-42c8-aaea-c82942348bf7-config\") pod \"service-ca-operator-57cdc4b768-nn5f8\" (UID: \"a1e60166-09e7-42c8-aaea-c82942348bf7\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" Sep 19 10:35:08.427251 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:08.427441 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.427441 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkt9\" (UniqueName: \"kubernetes.io/projected/bdae529f-3e44-4813-906d-fafa5aa85f27-kube-api-access-ggkt9\") pod \"kube-storage-version-migrator-operator-55889f7b47-2jc62\" (UID: \"bdae529f-3e44-4813-906d-fafa5aa85f27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" Sep 19 10:35:08.427441 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:08.427327 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 19 10:35:08.427441 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hphhx\" (UniqueName: \"kubernetes.io/projected/245f29a4-837b-4197-9eb7-52f324d83901-kube-api-access-hphhx\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:08.427441 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:08.427379 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls podName:245f29a4-837b-4197-9eb7-52f324d83901 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:08.927364546 +0000 UTC m=+40.839869263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-9rtmm" (UID: "245f29a4-837b-4197-9eb7-52f324d83901") : secret "cluster-monitoring-operator-tls" not found Sep 19 10:35:08.427441 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1e60166-09e7-42c8-aaea-c82942348bf7-serving-cert\") pod \"service-ca-operator-57cdc4b768-nn5f8\" (UID: \"a1e60166-09e7-42c8-aaea-c82942348bf7\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" Sep 19 10:35:08.427441 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427437 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efe9b96b-1362-4fc4-8701-36225c008ec9-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.427769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/efe9b96b-1362-4fc4-8701-36225c008ec9-serving-cert\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.427769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427514 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/efe9b96b-1362-4fc4-8701-36225c008ec9-snapshots\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.427769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lnf2\" (UniqueName: \"kubernetes.io/projected/efe9b96b-1362-4fc4-8701-36225c008ec9-kube-api-access-8lnf2\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.427769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e1e97edd-62b0-443a-b1cc-d7b796704644-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7jsd8\" (UID: \"e1e97edd-62b0-443a-b1cc-d7b796704644\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:08.427935 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.427889 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/245f29a4-837b-4197-9eb7-52f324d83901-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:08.429045 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.429023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bdae529f-3e44-4813-906d-fafa5aa85f27-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-2jc62\" (UID: \"bdae529f-3e44-4813-906d-fafa5aa85f27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" Sep 19 10:35:08.429606 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.429586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1e60166-09e7-42c8-aaea-c82942348bf7-serving-cert\") pod \"service-ca-operator-57cdc4b768-nn5f8\" (UID: \"a1e60166-09e7-42c8-aaea-c82942348bf7\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" Sep 19 10:35:08.435784 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.435720 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd29j\" (UniqueName: \"kubernetes.io/projected/c77ef5c2-0b57-4986-b196-8f007104d157-kube-api-access-kd29j\") pod \"network-check-source-7cbc45fddc-wxvs7\" (UID: \"c77ef5c2-0b57-4986-b196-8f007104d157\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-wxvs7" Sep 19 10:35:08.435995 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.435976 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkt9\" (UniqueName: \"kubernetes.io/projected/bdae529f-3e44-4813-906d-fafa5aa85f27-kube-api-access-ggkt9\") pod \"kube-storage-version-migrator-operator-55889f7b47-2jc62\" (UID: \"bdae529f-3e44-4813-906d-fafa5aa85f27\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" Sep 19 10:35:08.436173 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.436150 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5tsz\" (UniqueName: \"kubernetes.io/projected/a1e60166-09e7-42c8-aaea-c82942348bf7-kube-api-access-m5tsz\") pod \"service-ca-operator-57cdc4b768-nn5f8\" (UID: \"a1e60166-09e7-42c8-aaea-c82942348bf7\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" Sep 19 10:35:08.436418 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.436401 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hphhx\" (UniqueName: \"kubernetes.io/projected/245f29a4-837b-4197-9eb7-52f324d83901-kube-api-access-hphhx\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:08.446343 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.446323 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" Sep 19 10:35:08.468175 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.468150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" Sep 19 10:35:08.497434 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.497338 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jrfz" event={"ID":"5bcf3b15-117f-4aaf-a94a-08997ee640c8","Type":"ContainerStarted","Data":"b1537210ae52ee116cfc741d3c42e7783bdb9d5c32484f830141b75f42ba8d4a"} Sep 19 10:35:08.499391 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.499343 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4cgqn" event={"ID":"c425dda8-86ce-4a68-947b-e32ded00bfba","Type":"ContainerStarted","Data":"5ec56f1ff148db5e6b0d3c6f0b8f5d4376608bc4bea88b16446891e94fcd02a4"} Sep 19 10:35:08.514074 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.514034 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-wxvs7" Sep 19 10:35:08.522661 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.520628 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2jrfz" podStartSLOduration=9.254281041 podStartE2EDuration="38.520607548s" totalImagesPullingTime="29.266326507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:34:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:08.519568702 +0000 UTC m=+40.432073425" watchObservedRunningTime="2026-09-19 10:35:08.520607548 +0000 UTC m=+40.433112266" Sep 19 10:35:08.531441 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.531416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-default-certificate\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.531569 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.531459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5b9q\" (UniqueName: \"kubernetes.io/projected/6b92f4b1-f19a-42ac-b87b-835f23d214cd-kube-api-access-m5b9q\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.531569 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.531515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.531686 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.531576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efe9b96b-1362-4fc4-8701-36225c008ec9-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.531686 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.531606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/efe9b96b-1362-4fc4-8701-36225c008ec9-serving-cert\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.531686 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.531652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/efe9b96b-1362-4fc4-8701-36225c008ec9-snapshots\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.531686 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.531675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lnf2\" (UniqueName: \"kubernetes.io/projected/efe9b96b-1362-4fc4-8701-36225c008ec9-kube-api-access-8lnf2\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.531875 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.531699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efe9b96b-1362-4fc4-8701-36225c008ec9-tmp\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.531875 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.531725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtdw8\" (UniqueName: \"kubernetes.io/projected/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-kube-api-access-rtdw8\") pod \"cluster-samples-operator-685d7d68b5-d49c9\" (UID: \"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" Sep 19 10:35:08.531875 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.531774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efe9b96b-1362-4fc4-8701-36225c008ec9-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.531875 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.531799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-stats-auth\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.531875 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.531842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.531875 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.531869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d49c9\" (UID: \"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" Sep 19 10:35:08.532344 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:08.532003 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 19 10:35:08.532344 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:08.532084 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls podName:8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a nodeName:}" failed. No retries permitted until 2026-09-19 10:35:09.0320475 +0000 UTC m=+40.944552212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-d49c9" (UID: "8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a") : secret "samples-operator-tls" not found Sep 19 10:35:08.533928 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.533901 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efe9b96b-1362-4fc4-8701-36225c008ec9-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.534209 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.534186 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/efe9b96b-1362-4fc4-8701-36225c008ec9-snapshots\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.534405 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:08.534387 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 10:35:08.534611 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:08.534598 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs podName:6b92f4b1-f19a-42ac-b87b-835f23d214cd nodeName:}" failed. No retries permitted until 2026-09-19 10:35:09.034576412 +0000 UTC m=+40.947081114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs") pod "router-default-9998785cb-l2lkw" (UID: "6b92f4b1-f19a-42ac-b87b-835f23d214cd") : secret "router-metrics-certs-default" not found Sep 19 10:35:08.534773 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:08.534751 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle podName:6b92f4b1-f19a-42ac-b87b-835f23d214cd nodeName:}" failed. No retries permitted until 2026-09-19 10:35:09.034734778 +0000 UTC m=+40.947239478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle") pod "router-default-9998785cb-l2lkw" (UID: "6b92f4b1-f19a-42ac-b87b-835f23d214cd") : configmap references non-existent config key: service-ca.crt Sep 19 10:35:08.534773 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.534643 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efe9b96b-1362-4fc4-8701-36225c008ec9-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.534899 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.534856 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efe9b96b-1362-4fc4-8701-36225c008ec9-tmp\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.536281 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.536231 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4cgqn" podStartSLOduration=2.5512223450000002 podStartE2EDuration="5.53621603s" totalImagesPullingTime="2.984993685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:08.535556918 +0000 UTC m=+40.448061638" watchObservedRunningTime="2026-09-19 10:35:08.53621603 +0000 UTC m=+40.448720747" Sep 19 10:35:08.537886 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.537859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-default-certificate\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.538201 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.538180 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/efe9b96b-1362-4fc4-8701-36225c008ec9-serving-cert\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.539273 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.539249 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-stats-auth\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.543584 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.543538 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtdw8\" (UniqueName: \"kubernetes.io/projected/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-kube-api-access-rtdw8\") pod \"cluster-samples-operator-685d7d68b5-d49c9\" (UID: \"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" Sep 19 10:35:08.543584 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.543553 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lnf2\" (UniqueName: \"kubernetes.io/projected/efe9b96b-1362-4fc4-8701-36225c008ec9-kube-api-access-8lnf2\") pod \"insights-operator-6f9c644dd9-t5cft\" (UID: \"efe9b96b-1362-4fc4-8701-36225c008ec9\") " pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.545898 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.545880 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5b9q\" (UniqueName: \"kubernetes.io/projected/6b92f4b1-f19a-42ac-b87b-835f23d214cd-kube-api-access-m5b9q\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:08.601098 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.600746 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" Sep 19 10:35:08.697801 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.697774 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4md95"] Sep 19 10:35:08.712885 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:08.712860 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697fd709_5743_4f20_aa9b_8cc6285edeaa.slice/crio-03552528c07ac195fda521ed7437d1fbff39702471102445ded9881275f97166 WatchSource:0}: Error finding container 03552528c07ac195fda521ed7437d1fbff39702471102445ded9881275f97166: Status 404 returned error can't find the container with id 03552528c07ac195fda521ed7437d1fbff39702471102445ded9881275f97166 Sep 19 10:35:08.746482 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.746422 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-xnm4h"] Sep 19 10:35:08.748816 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:08.748791 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61826a91_39c7_4d1b_a0f9_c1de1de49c0e.slice/crio-9ab64df5b51082255ef711f5e34dcda10fe01abbbdc1895b648ea06f1159ff06 WatchSource:0}: Error finding container 9ab64df5b51082255ef711f5e34dcda10fe01abbbdc1895b648ea06f1159ff06: Status 404 returned error can't find the container with id 9ab64df5b51082255ef711f5e34dcda10fe01abbbdc1895b648ea06f1159ff06 Sep 19 10:35:08.776357 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.776334 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62"] Sep 19 10:35:08.779784 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:08.779765 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdae529f_3e44_4813_906d_fafa5aa85f27.slice/crio-a6d88b734d8b6ceb28c4b83e5255ceb6aac481ad404ead1caa76e7462cac3930 WatchSource:0}: Error finding container a6d88b734d8b6ceb28c4b83e5255ceb6aac481ad404ead1caa76e7462cac3930: Status 404 returned error can't find the container with id a6d88b734d8b6ceb28c4b83e5255ceb6aac481ad404ead1caa76e7462cac3930 Sep 19 10:35:08.810700 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.810679 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8"] Sep 19 10:35:08.812850 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:08.812825 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e60166_09e7_42c8_aaea_c82942348bf7.slice/crio-75c1bd46b37fe38fb25ed30bcb53dd710295e41c8d83c897873071f0b0cdd14f WatchSource:0}: Error finding container 75c1bd46b37fe38fb25ed30bcb53dd710295e41c8d83c897873071f0b0cdd14f: Status 404 returned error can't find the container with id 75c1bd46b37fe38fb25ed30bcb53dd710295e41c8d83c897873071f0b0cdd14f Sep 19 10:35:08.831972 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.831951 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-wxvs7"] Sep 19 10:35:08.834135 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:08.834114 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77ef5c2_0b57_4986_b196_8f007104d157.slice/crio-a052ca64cca384a3cf6642ed2a5418b577ef83a0ae8a9ced738b061315d2eda4 WatchSource:0}: Error finding container a052ca64cca384a3cf6642ed2a5418b577ef83a0ae8a9ced738b061315d2eda4: Status 404 returned error can't find the container with id a052ca64cca384a3cf6642ed2a5418b577ef83a0ae8a9ced738b061315d2eda4 Sep 19 10:35:08.909686 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.909662 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-t5cft"] Sep 19 10:35:08.911911 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:08.911883 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe9b96b_1362_4fc4_8701_36225c008ec9.slice/crio-e0a030cd522c33d0a1d8212e9e6f553d031d4666e95fc40575e3cf00e84caea7 WatchSource:0}: Error finding container e0a030cd522c33d0a1d8212e9e6f553d031d4666e95fc40575e3cf00e84caea7: Status 404 returned error can't find the container with id e0a030cd522c33d0a1d8212e9e6f553d031d4666e95fc40575e3cf00e84caea7 Sep 19 10:35:08.934814 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.934795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jsd8\" (UID: \"e1e97edd-62b0-443a-b1cc-d7b796704644\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:08.934909 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:08.934871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:08.934961 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:08.934936 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 19 10:35:08.935008 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:08.934991 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert podName:e1e97edd-62b0-443a-b1cc-d7b796704644 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:09.934976472 +0000 UTC m=+41.847481183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7jsd8" (UID: "e1e97edd-62b0-443a-b1cc-d7b796704644") : secret "networking-console-plugin-cert" not found Sep 19 10:35:08.935089 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:08.935021 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 19 10:35:08.935133 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:08.935096 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls podName:245f29a4-837b-4197-9eb7-52f324d83901 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:09.935076974 +0000 UTC m=+41.847581675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-9rtmm" (UID: "245f29a4-837b-4197-9eb7-52f324d83901") : secret "cluster-monitoring-operator-tls" not found Sep 19 10:35:09.035791 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:09.035731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:09.035791 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:09.035763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d49c9\" (UID: \"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" Sep 19 10:35:09.035939 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:09.035826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:09.035939 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:09.035891 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle podName:6b92f4b1-f19a-42ac-b87b-835f23d214cd nodeName:}" failed. No retries permitted until 2026-09-19 10:35:10.03587338 +0000 UTC m=+41.948378146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle") pod "router-default-9998785cb-l2lkw" (UID: "6b92f4b1-f19a-42ac-b87b-835f23d214cd") : configmap references non-existent config key: service-ca.crt Sep 19 10:35:09.036043 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:09.035956 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 10:35:09.036043 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:09.035960 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 19 10:35:09.036043 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:09.036003 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs podName:6b92f4b1-f19a-42ac-b87b-835f23d214cd nodeName:}" failed. No retries permitted until 2026-09-19 10:35:10.035989837 +0000 UTC m=+41.948494545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs") pod "router-default-9998785cb-l2lkw" (UID: "6b92f4b1-f19a-42ac-b87b-835f23d214cd") : secret "router-metrics-certs-default" not found Sep 19 10:35:09.036043 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:09.036019 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls podName:8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a nodeName:}" failed. No retries permitted until 2026-09-19 10:35:10.036010384 +0000 UTC m=+41.948515079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-d49c9" (UID: "8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a") : secret "samples-operator-tls" not found Sep 19 10:35:09.503347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:09.503309 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" event={"ID":"61826a91-39c7-4d1b-a0f9-c1de1de49c0e","Type":"ContainerStarted","Data":"9ab64df5b51082255ef711f5e34dcda10fe01abbbdc1895b648ea06f1159ff06"} Sep 19 10:35:09.505170 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:09.505000 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" event={"ID":"efe9b96b-1362-4fc4-8701-36225c008ec9","Type":"ContainerStarted","Data":"e0a030cd522c33d0a1d8212e9e6f553d031d4666e95fc40575e3cf00e84caea7"} Sep 19 10:35:09.506660 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:09.506633 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-wxvs7" event={"ID":"c77ef5c2-0b57-4986-b196-8f007104d157","Type":"ContainerStarted","Data":"a052ca64cca384a3cf6642ed2a5418b577ef83a0ae8a9ced738b061315d2eda4"} Sep 19 10:35:09.508574 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:09.508542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" event={"ID":"a1e60166-09e7-42c8-aaea-c82942348bf7","Type":"ContainerStarted","Data":"75c1bd46b37fe38fb25ed30bcb53dd710295e41c8d83c897873071f0b0cdd14f"} Sep 19 10:35:09.511153 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:09.511127 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" event={"ID":"bdae529f-3e44-4813-906d-fafa5aa85f27","Type":"ContainerStarted","Data":"a6d88b734d8b6ceb28c4b83e5255ceb6aac481ad404ead1caa76e7462cac3930"} Sep 19 10:35:09.513797 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:09.513774 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4md95" event={"ID":"697fd709-5743-4f20-aa9b-8cc6285edeaa","Type":"ContainerStarted","Data":"03552528c07ac195fda521ed7437d1fbff39702471102445ded9881275f97166"} Sep 19 10:35:09.945834 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:09.945421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:09.945834 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:09.945571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jsd8\" (UID: \"e1e97edd-62b0-443a-b1cc-d7b796704644\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:09.945834 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:09.945594 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 19 10:35:09.945834 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:09.945677 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls podName:245f29a4-837b-4197-9eb7-52f324d83901 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:11.945655915 +0000 UTC m=+43.858160612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-9rtmm" (UID: "245f29a4-837b-4197-9eb7-52f324d83901") : secret "cluster-monitoring-operator-tls" not found Sep 19 10:35:09.945834 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:09.945734 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 19 10:35:09.945834 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:09.945785 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert podName:e1e97edd-62b0-443a-b1cc-d7b796704644 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:11.945769612 +0000 UTC m=+43.858274319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7jsd8" (UID: "e1e97edd-62b0-443a-b1cc-d7b796704644") : secret "networking-console-plugin-cert" not found Sep 19 10:35:10.047286 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:10.046857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:10.047286 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:10.046998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:10.047286 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:10.047031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d49c9\" (UID: \"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" Sep 19 10:35:10.047286 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:10.047174 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 19 10:35:10.047286 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:10.047208 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 10:35:10.047286 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:10.047231 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls podName:8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a nodeName:}" failed. No retries permitted until 2026-09-19 10:35:12.047213054 +0000 UTC m=+43.959717756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-d49c9" (UID: "8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a") : secret "samples-operator-tls" not found Sep 19 10:35:10.047286 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:10.047249 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle podName:6b92f4b1-f19a-42ac-b87b-835f23d214cd nodeName:}" failed. No retries permitted until 2026-09-19 10:35:12.04723961 +0000 UTC m=+43.959744313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle") pod "router-default-9998785cb-l2lkw" (UID: "6b92f4b1-f19a-42ac-b87b-835f23d214cd") : configmap references non-existent config key: service-ca.crt Sep 19 10:35:10.047286 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:10.047264 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs podName:6b92f4b1-f19a-42ac-b87b-835f23d214cd nodeName:}" failed. No retries permitted until 2026-09-19 10:35:12.047255719 +0000 UTC m=+43.959760418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs") pod "router-default-9998785cb-l2lkw" (UID: "6b92f4b1-f19a-42ac-b87b-835f23d214cd") : secret "router-metrics-certs-default" not found Sep 19 10:35:10.519914 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:10.519826 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4md95" event={"ID":"697fd709-5743-4f20-aa9b-8cc6285edeaa","Type":"ContainerStarted","Data":"3de077d12751fbbea1a4b379a21372ddbed41c8c2a1ff3d0f9e54b33ec334552"} Sep 19 10:35:11.562514 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:11.562481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:11.562942 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:11.562633 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 10:35:11.562942 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:11.562668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:11.562942 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:11.562714 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls podName:6a046e7f-51bb-42f3-be5a-e3f15a80afce nodeName:}" failed. No retries permitted until 2026-09-19 10:35:19.562693216 +0000 UTC m=+51.475197914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls") pod "dns-default-gvn74" (UID: "6a046e7f-51bb-42f3-be5a-e3f15a80afce") : secret "dns-default-metrics-tls" not found Sep 19 10:35:11.562942 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:11.562777 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 19 10:35:11.562942 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:11.562791 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7774cfc878-x5gcd: secret "image-registry-tls" not found Sep 19 10:35:11.562942 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:11.562837 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls podName:13a1b6fa-ab80-4027-981f-5daf44a44b61 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:19.5628261 +0000 UTC m=+51.475330807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls") pod "image-registry-7774cfc878-x5gcd" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61") : secret "image-registry-tls" not found Sep 19 10:35:11.964532 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:11.964489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jsd8\" (UID: \"e1e97edd-62b0-443a-b1cc-d7b796704644\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:11.964713 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:11.964662 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 19 10:35:11.964795 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:11.964741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:11.964848 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:11.964824 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 19 10:35:11.964848 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:11.964841 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert podName:e1e97edd-62b0-443a-b1cc-d7b796704644 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:15.964819807 +0000 UTC m=+47.877324509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7jsd8" (UID: "e1e97edd-62b0-443a-b1cc-d7b796704644") : secret "networking-console-plugin-cert" not found Sep 19 10:35:11.964943 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:11.964880 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls podName:245f29a4-837b-4197-9eb7-52f324d83901 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:15.964868322 +0000 UTC m=+47.877373031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-9rtmm" (UID: "245f29a4-837b-4197-9eb7-52f324d83901") : secret "cluster-monitoring-operator-tls" not found Sep 19 10:35:12.065859 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:12.065826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:12.066051 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:12.065952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:12.066051 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:12.065981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d49c9\" (UID: \"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" Sep 19 10:35:12.066051 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:12.066000 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 10:35:12.066206 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:12.066091 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs podName:6b92f4b1-f19a-42ac-b87b-835f23d214cd nodeName:}" failed. No retries permitted until 2026-09-19 10:35:16.066070858 +0000 UTC m=+47.978575569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs") pod "router-default-9998785cb-l2lkw" (UID: "6b92f4b1-f19a-42ac-b87b-835f23d214cd") : secret "router-metrics-certs-default" not found Sep 19 10:35:12.066206 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:12.066110 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle podName:6b92f4b1-f19a-42ac-b87b-835f23d214cd nodeName:}" failed. No retries permitted until 2026-09-19 10:35:16.066100808 +0000 UTC m=+47.978605506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle") pod "router-default-9998785cb-l2lkw" (UID: "6b92f4b1-f19a-42ac-b87b-835f23d214cd") : configmap references non-existent config key: service-ca.crt Sep 19 10:35:12.066206 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:12.066118 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 19 10:35:12.066206 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:12.066170 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls podName:8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a nodeName:}" failed. No retries permitted until 2026-09-19 10:35:16.066155679 +0000 UTC m=+47.978660392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-d49c9" (UID: "8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a") : secret "samples-operator-tls" not found Sep 19 10:35:14.534136 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:14.534100 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-wxvs7" event={"ID":"c77ef5c2-0b57-4986-b196-8f007104d157","Type":"ContainerStarted","Data":"bf993fbd07b9983deea91b4c6dee2a9deae2d1837f34e2c7c5f434207903fd0e"} Sep 19 10:35:14.535999 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:14.535966 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" event={"ID":"a1e60166-09e7-42c8-aaea-c82942348bf7","Type":"ContainerStarted","Data":"df69cd9d9d30480e4bb4f7c7d7cdd6a2d45473a1fbf95c356b8c71f35b8e1198"} Sep 19 10:35:14.537746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:14.537719 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" event={"ID":"bdae529f-3e44-4813-906d-fafa5aa85f27","Type":"ContainerStarted","Data":"68ce6f0b0f819e5720cbbace3139585e054a7a26a5cc62f9411b7ace86afb873"} Sep 19 10:35:14.539568 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:14.539546 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/0.log" Sep 19 10:35:14.539675 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:14.539587 2579 generic.go:356] "Generic (PLEG): container finished" podID="61826a91-39c7-4d1b-a0f9-c1de1de49c0e" containerID="38cdd39841cd3ae590a3604e31c8221f4a06088fdf2659b4200396b774ade0e6" exitCode=255 Sep 19 10:35:14.539795 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:14.539769 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" event={"ID":"61826a91-39c7-4d1b-a0f9-c1de1de49c0e","Type":"ContainerDied","Data":"38cdd39841cd3ae590a3604e31c8221f4a06088fdf2659b4200396b774ade0e6"} Sep 19 10:35:14.539924 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:14.539882 2579 scope.go:117] "RemoveContainer" containerID="38cdd39841cd3ae590a3604e31c8221f4a06088fdf2659b4200396b774ade0e6" Sep 19 10:35:14.541394 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:14.541369 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" event={"ID":"efe9b96b-1362-4fc4-8701-36225c008ec9","Type":"ContainerStarted","Data":"56788311fc6e6fff0422e18c836a1a94333e64f0247cd52a415dbd98fd93d1f0"} Sep 19 10:35:14.549314 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:14.549272 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4md95" podStartSLOduration=5.084864881 podStartE2EDuration="6.549254532s" totalImagesPullingTime="1.464389651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:10.548130415 +0000 UTC m=+42.460635135" watchObservedRunningTime="2026-09-19 10:35:14.549254532 +0000 UTC m=+46.461759250" Sep 19 10:35:14.569378 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:14.569343 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-wxvs7" podStartSLOduration=1.599769097 podStartE2EDuration="6.569330363s" totalImagesPullingTime="4.969561266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:14.550387034 +0000 UTC m=+46.462891753" watchObservedRunningTime="2026-09-19 10:35:14.569330363 +0000 UTC m=+46.481835104" Sep 19 10:35:14.577235 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:14.577191 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" podStartSLOduration=1.692394953 podStartE2EDuration="6.576000727s" totalImagesPullingTime="4.883605774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:14.568945049 +0000 UTC m=+46.481449767" watchObservedRunningTime="2026-09-19 10:35:14.576000727 +0000 UTC m=+46.488505436" Sep 19 10:35:14.644894 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:14.644069 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" podStartSLOduration=1.6316901289999999 podStartE2EDuration="6.644033501s" totalImagesPullingTime="5.012343372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:14.618933179 +0000 UTC m=+46.531437900" watchObservedRunningTime="2026-09-19 10:35:14.644033501 +0000 UTC m=+46.556538218" Sep 19 10:35:14.645289 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:14.645239 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" podStartSLOduration=1.663776703 podStartE2EDuration="6.645226304s" totalImagesPullingTime="4.981449601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:14.643077417 +0000 UTC m=+46.555582243" watchObservedRunningTime="2026-09-19 10:35:14.645226304 +0000 UTC m=+46.557731048" Sep 19 10:35:15.514987 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.514946 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pq7d5"] Sep 19 10:35:15.541104 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.541054 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pq7d5"] Sep 19 10:35:15.541506 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.541150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:15.543650 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.543605 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 19 10:35:15.543934 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.543919 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rcqd6\"" Sep 19 10:35:15.544025 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.543919 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 19 10:35:15.546342 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.546322 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/1.log" Sep 19 10:35:15.546743 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.546729 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/0.log" Sep 19 10:35:15.546824 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.546760 2579 generic.go:356] "Generic (PLEG): container finished" podID="61826a91-39c7-4d1b-a0f9-c1de1de49c0e" containerID="772e38f3474cee6ed1634fc47074f05943211266ad24cd53db9592190f0216a6" exitCode=255 Sep 19 10:35:15.546884 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.546830 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" event={"ID":"61826a91-39c7-4d1b-a0f9-c1de1de49c0e","Type":"ContainerDied","Data":"772e38f3474cee6ed1634fc47074f05943211266ad24cd53db9592190f0216a6"} Sep 19 10:35:15.546932 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.546895 2579 scope.go:117] "RemoveContainer" containerID="38cdd39841cd3ae590a3604e31c8221f4a06088fdf2659b4200396b774ade0e6" Sep 19 10:35:15.547171 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.547151 2579 scope.go:117] "RemoveContainer" containerID="772e38f3474cee6ed1634fc47074f05943211266ad24cd53db9592190f0216a6" Sep 19 10:35:15.547385 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:15.547360 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-xnm4h_openshift-console-operator(61826a91-39c7-4d1b-a0f9-c1de1de49c0e)\"" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" podUID="61826a91-39c7-4d1b-a0f9-c1de1de49c0e" Sep 19 10:35:15.605359 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.605335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-crio-socket\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:15.605535 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.605512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrcgn\" (UniqueName: \"kubernetes.io/projected/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-api-access-nrcgn\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:15.605767 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.605745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-data-volume\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:15.606133 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.605910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:15.606133 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.606006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:15.706521 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.706489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-crio-socket\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:15.706657 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.706535 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrcgn\" (UniqueName: \"kubernetes.io/projected/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-api-access-nrcgn\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:15.706702 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.706679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-data-volume\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:15.706702 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.706689 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-crio-socket\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:15.706775 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.706745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:15.706817 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.706793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:15.706882 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:15.706869 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 19 10:35:15.706945 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:15.706891 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 10:35:15.706945 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.706911 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-data-volume\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:15.706945 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:15.706924 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm podName:2f036b1a-5177-4773-b2d1-d2537cf6d5e1 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:16.206911307 +0000 UTC m=+48.119416014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pq7d5" (UID: "2f036b1a-5177-4773-b2d1-d2537cf6d5e1") : configmap "kube-rbac-proxy" not found Sep 19 10:35:15.707044 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:15.706953 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls podName:2f036b1a-5177-4773-b2d1-d2537cf6d5e1 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:16.20693134 +0000 UTC m=+48.119436047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pq7d5" (UID: "2f036b1a-5177-4773-b2d1-d2537cf6d5e1") : secret "insights-runtime-extractor-tls" not found Sep 19 10:35:15.715931 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:15.715911 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrcgn\" (UniqueName: \"kubernetes.io/projected/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-api-access-nrcgn\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:16.009245 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:16.009209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jsd8\" (UID: \"e1e97edd-62b0-443a-b1cc-d7b796704644\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:16.009397 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:16.009307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:16.009397 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.009345 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 19 10:35:16.009397 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.009394 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 19 10:35:16.009501 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.009408 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert podName:e1e97edd-62b0-443a-b1cc-d7b796704644 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:24.009393736 +0000 UTC m=+55.921898429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7jsd8" (UID: "e1e97edd-62b0-443a-b1cc-d7b796704644") : secret "networking-console-plugin-cert" not found Sep 19 10:35:16.009501 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.009432 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls podName:245f29a4-837b-4197-9eb7-52f324d83901 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:24.009421401 +0000 UTC m=+55.921926117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-9rtmm" (UID: "245f29a4-837b-4197-9eb7-52f324d83901") : secret "cluster-monitoring-operator-tls" not found Sep 19 10:35:16.110575 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:16.110550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:16.110705 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:16.110585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d49c9\" (UID: \"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" Sep 19 10:35:16.110705 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:16.110659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:16.110705 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.110679 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 19 10:35:16.110849 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.110717 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls podName:8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a nodeName:}" failed. No retries permitted until 2026-09-19 10:35:24.110705543 +0000 UTC m=+56.023210237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-d49c9" (UID: "8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a") : secret "samples-operator-tls" not found Sep 19 10:35:16.110849 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.110756 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 10:35:16.110849 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.110785 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle podName:6b92f4b1-f19a-42ac-b87b-835f23d214cd nodeName:}" failed. No retries permitted until 2026-09-19 10:35:24.110767219 +0000 UTC m=+56.023271928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle") pod "router-default-9998785cb-l2lkw" (UID: "6b92f4b1-f19a-42ac-b87b-835f23d214cd") : configmap references non-existent config key: service-ca.crt Sep 19 10:35:16.110849 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.110811 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs podName:6b92f4b1-f19a-42ac-b87b-835f23d214cd nodeName:}" failed. No retries permitted until 2026-09-19 10:35:24.110800269 +0000 UTC m=+56.023304971 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs") pod "router-default-9998785cb-l2lkw" (UID: "6b92f4b1-f19a-42ac-b87b-835f23d214cd") : secret "router-metrics-certs-default" not found Sep 19 10:35:16.211234 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:16.211209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:16.211382 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:16.211256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:16.211382 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.211311 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 19 10:35:16.211382 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.211346 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm podName:2f036b1a-5177-4773-b2d1-d2537cf6d5e1 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:17.211336036 +0000 UTC m=+49.123840730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pq7d5" (UID: "2f036b1a-5177-4773-b2d1-d2537cf6d5e1") : configmap "kube-rbac-proxy" not found Sep 19 10:35:16.211382 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.211348 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 10:35:16.211576 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.211411 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls podName:2f036b1a-5177-4773-b2d1-d2537cf6d5e1 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:17.211397699 +0000 UTC m=+49.123902411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pq7d5" (UID: "2f036b1a-5177-4773-b2d1-d2537cf6d5e1") : secret "insights-runtime-extractor-tls" not found Sep 19 10:35:16.552116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:16.552085 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/1.log" Sep 19 10:35:16.552501 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:16.552465 2579 scope.go:117] "RemoveContainer" containerID="772e38f3474cee6ed1634fc47074f05943211266ad24cd53db9592190f0216a6" Sep 19 10:35:16.552632 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:16.552614 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-xnm4h_openshift-console-operator(61826a91-39c7-4d1b-a0f9-c1de1de49c0e)\"" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" podUID="61826a91-39c7-4d1b-a0f9-c1de1de49c0e" Sep 19 10:35:17.221224 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:17.221186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:17.221363 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:17.221240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:17.221363 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:17.221341 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 10:35:17.221439 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:17.221373 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 19 10:35:17.221439 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:17.221406 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls podName:2f036b1a-5177-4773-b2d1-d2537cf6d5e1 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:19.221390898 +0000 UTC m=+51.133895614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pq7d5" (UID: "2f036b1a-5177-4773-b2d1-d2537cf6d5e1") : secret "insights-runtime-extractor-tls" not found Sep 19 10:35:17.221439 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:17.221426 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm podName:2f036b1a-5177-4773-b2d1-d2537cf6d5e1 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:19.221413555 +0000 UTC m=+51.133918250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pq7d5" (UID: "2f036b1a-5177-4773-b2d1-d2537cf6d5e1") : configmap "kube-rbac-proxy" not found Sep 19 10:35:17.292879 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:17.292856 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c9c22_9e5b260f-bb0a-4096-9a9f-f4a534e77eea/dns-node-resolver/0.log" Sep 19 10:35:18.295944 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:18.295919 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k2ddk_747ac66b-6228-471f-9409-77c6246911dd/node-ca/0.log" Sep 19 10:35:18.423906 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:18.423873 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:18.424078 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:18.423924 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:18.424257 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:18.424246 2579 scope.go:117] "RemoveContainer" containerID="772e38f3474cee6ed1634fc47074f05943211266ad24cd53db9592190f0216a6" Sep 19 10:35:18.424401 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:18.424384 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-xnm4h_openshift-console-operator(61826a91-39c7-4d1b-a0f9-c1de1de49c0e)\"" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" podUID="61826a91-39c7-4d1b-a0f9-c1de1de49c0e" Sep 19 10:35:19.094589 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:19.094543 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4cgqn_c425dda8-86ce-4a68-947b-e32ded00bfba/serve-healthcheck-canary/0.log" Sep 19 10:35:19.240113 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:19.240052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:19.240291 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:19.240156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:19.240291 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:19.240215 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 10:35:19.240291 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:19.240253 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 19 10:35:19.240394 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:19.240293 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm podName:2f036b1a-5177-4773-b2d1-d2537cf6d5e1 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:23.240276571 +0000 UTC m=+55.152781267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pq7d5" (UID: "2f036b1a-5177-4773-b2d1-d2537cf6d5e1") : configmap "kube-rbac-proxy" not found Sep 19 10:35:19.240394 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:19.240317 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls podName:2f036b1a-5177-4773-b2d1-d2537cf6d5e1 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:23.24030846 +0000 UTC m=+55.152813159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pq7d5" (UID: "2f036b1a-5177-4773-b2d1-d2537cf6d5e1") : secret "insights-runtime-extractor-tls" not found Sep 19 10:35:19.644035 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:19.643999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:19.644493 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:19.644046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:19.644493 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:19.644175 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 10:35:19.644493 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:19.644191 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 19 10:35:19.644493 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:19.644211 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7774cfc878-x5gcd: secret "image-registry-tls" not found Sep 19 10:35:19.644493 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:19.644223 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls podName:6a046e7f-51bb-42f3-be5a-e3f15a80afce nodeName:}" failed. No retries permitted until 2026-09-19 10:35:35.64421054 +0000 UTC m=+67.556715234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls") pod "dns-default-gvn74" (UID: "6a046e7f-51bb-42f3-be5a-e3f15a80afce") : secret "dns-default-metrics-tls" not found Sep 19 10:35:19.644493 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:19.644265 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls podName:13a1b6fa-ab80-4027-981f-5daf44a44b61 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:35.644246646 +0000 UTC m=+67.556751353 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls") pod "image-registry-7774cfc878-x5gcd" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61") : secret "image-registry-tls" not found Sep 19 10:35:23.274015 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:23.273975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:23.274514 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:23.274038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:23.274514 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:23.274149 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 10:35:23.274514 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:23.274209 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls podName:2f036b1a-5177-4773-b2d1-d2537cf6d5e1 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:31.274193217 +0000 UTC m=+63.186697928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pq7d5" (UID: "2f036b1a-5177-4773-b2d1-d2537cf6d5e1") : secret "insights-runtime-extractor-tls" not found Sep 19 10:35:23.274698 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:23.274643 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:24.081103 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:24.081050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:24.081288 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:24.081157 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jsd8\" (UID: \"e1e97edd-62b0-443a-b1cc-d7b796704644\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:24.081288 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:24.081209 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 19 10:35:24.081288 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:24.081272 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 19 10:35:24.081288 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:24.081286 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls podName:245f29a4-837b-4197-9eb7-52f324d83901 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:40.081270034 +0000 UTC m=+71.993774750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-9rtmm" (UID: "245f29a4-837b-4197-9eb7-52f324d83901") : secret "cluster-monitoring-operator-tls" not found Sep 19 10:35:24.081443 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:24.081312 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert podName:e1e97edd-62b0-443a-b1cc-d7b796704644 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:40.081301108 +0000 UTC m=+71.993805802 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7jsd8" (UID: "e1e97edd-62b0-443a-b1cc-d7b796704644") : secret "networking-console-plugin-cert" not found Sep 19 10:35:24.182497 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:24.182459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:24.182660 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:24.182507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d49c9\" (UID: \"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" Sep 19 10:35:24.182660 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:24.182554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:24.182660 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:24.182615 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle podName:6b92f4b1-f19a-42ac-b87b-835f23d214cd nodeName:}" failed. No retries permitted until 2026-09-19 10:35:40.182595351 +0000 UTC m=+72.095100046 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle") pod "router-default-9998785cb-l2lkw" (UID: "6b92f4b1-f19a-42ac-b87b-835f23d214cd") : configmap references non-existent config key: service-ca.crt Sep 19 10:35:24.182660 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:24.182656 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 10:35:24.182812 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:24.182694 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs podName:6b92f4b1-f19a-42ac-b87b-835f23d214cd nodeName:}" failed. No retries permitted until 2026-09-19 10:35:40.182682336 +0000 UTC m=+72.095187030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs") pod "router-default-9998785cb-l2lkw" (UID: "6b92f4b1-f19a-42ac-b87b-835f23d214cd") : secret "router-metrics-certs-default" not found Sep 19 10:35:24.184886 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:24.184864 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-d49c9\" (UID: \"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" Sep 19 10:35:24.216894 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:24.216867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" Sep 19 10:35:24.545532 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:24.545507 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9"] Sep 19 10:35:25.582648 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:25.582618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" event={"ID":"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a","Type":"ContainerStarted","Data":"49930fa8e4554819b5e7643c214e506c2e6c50154a51511b36f5370c23a8d563"} Sep 19 10:35:27.592614 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:27.592573 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" event={"ID":"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a","Type":"ContainerStarted","Data":"5d605c0d79950599097680f83104510e73eceb6c0ff038f60506ccea3a485ea3"} Sep 19 10:35:27.593010 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:27.592673 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" event={"ID":"8a21c7e1-68d3-4d30-aaeb-ffdef769cb6a","Type":"ContainerStarted","Data":"2d997f55173812e12c187e169449b16d94579d980f6e7f237e1c0dd01f0e69ed"} Sep 19 10:35:27.609359 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:27.609291 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-d49c9" podStartSLOduration=17.540261165 podStartE2EDuration="19.609273329s" totalImagesPullingTime="2.069012164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:27.608555427 +0000 UTC m=+59.521060144" watchObservedRunningTime="2026-09-19 10:35:27.609273329 +0000 UTC m=+59.521778047" Sep 19 10:35:28.460390 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:28.460364 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jzdrv" Sep 19 10:35:31.354395 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:31.354360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:31.356536 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:31.356510 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f036b1a-5177-4773-b2d1-d2537cf6d5e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pq7d5\" (UID: \"2f036b1a-5177-4773-b2d1-d2537cf6d5e1\") " pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:31.454282 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:31.454254 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rcqd6\"" Sep 19 10:35:31.462537 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:31.462511 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pq7d5" Sep 19 10:35:31.769295 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:31.769266 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pq7d5"] Sep 19 10:35:31.771951 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:31.771927 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f036b1a_5177_4773_b2d1_d2537cf6d5e1.slice/crio-c1dcb77262fe6d6e36f6d566c89d54317e29ab9986de6263d2d8637819d4dd77 WatchSource:0}: Error finding container c1dcb77262fe6d6e36f6d566c89d54317e29ab9986de6263d2d8637819d4dd77: Status 404 returned error can't find the container with id c1dcb77262fe6d6e36f6d566c89d54317e29ab9986de6263d2d8637819d4dd77 Sep 19 10:35:32.190732 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:32.190705 2579 scope.go:117] "RemoveContainer" containerID="772e38f3474cee6ed1634fc47074f05943211266ad24cd53db9592190f0216a6" Sep 19 10:35:32.609223 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:32.609154 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pq7d5" event={"ID":"2f036b1a-5177-4773-b2d1-d2537cf6d5e1","Type":"ContainerStarted","Data":"aefab2ec0b0a9a289792056c92ab4361a759745b6c229bcf0dd67744322e7a9a"} Sep 19 10:35:32.609588 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:32.609222 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pq7d5" event={"ID":"2f036b1a-5177-4773-b2d1-d2537cf6d5e1","Type":"ContainerStarted","Data":"c1dcb77262fe6d6e36f6d566c89d54317e29ab9986de6263d2d8637819d4dd77"} Sep 19 10:35:32.610545 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:32.610525 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 10:35:32.610893 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:32.610877 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/1.log" Sep 19 10:35:32.610939 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:32.610912 2579 generic.go:356] "Generic (PLEG): container finished" podID="61826a91-39c7-4d1b-a0f9-c1de1de49c0e" containerID="318dc4f2782bcb9182240b9ebeabaa3ea56f4ab193f6ae8e456cd099aa77a0fa" exitCode=255 Sep 19 10:35:32.610974 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:32.610941 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" event={"ID":"61826a91-39c7-4d1b-a0f9-c1de1de49c0e","Type":"ContainerDied","Data":"318dc4f2782bcb9182240b9ebeabaa3ea56f4ab193f6ae8e456cd099aa77a0fa"} Sep 19 10:35:32.611011 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:32.610976 2579 scope.go:117] "RemoveContainer" containerID="772e38f3474cee6ed1634fc47074f05943211266ad24cd53db9592190f0216a6" Sep 19 10:35:32.611287 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:32.611266 2579 scope.go:117] "RemoveContainer" containerID="318dc4f2782bcb9182240b9ebeabaa3ea56f4ab193f6ae8e456cd099aa77a0fa" Sep 19 10:35:32.611468 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:32.611448 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-xnm4h_openshift-console-operator(61826a91-39c7-4d1b-a0f9-c1de1de49c0e)\"" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" podUID="61826a91-39c7-4d1b-a0f9-c1de1de49c0e" Sep 19 10:35:33.615882 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:33.615858 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 10:35:33.618125 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:33.618090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pq7d5" event={"ID":"2f036b1a-5177-4773-b2d1-d2537cf6d5e1","Type":"ContainerStarted","Data":"65d6e857077223131d9f314bb076ea7df493249550be171425d27ef0b7f9c7e7"} Sep 19 10:35:35.625577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.625544 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pq7d5" event={"ID":"2f036b1a-5177-4773-b2d1-d2537cf6d5e1","Type":"ContainerStarted","Data":"09fa72169eceb16d3e4d2254fd6cca2aad7598db90ce75eca6d4afc9c97ec649"} Sep 19 10:35:35.645792 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.645748 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pq7d5" podStartSLOduration=17.843485791 podStartE2EDuration="20.645734211s" totalImagesPullingTime="2.80224842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:35:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:35.645129137 +0000 UTC m=+67.557633853" watchObservedRunningTime="2026-09-19 10:35:35.645734211 +0000 UTC m=+67.558238927" Sep 19 10:35:35.690909 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.690881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:35.691018 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.690925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:35.693218 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.693193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a046e7f-51bb-42f3-be5a-e3f15a80afce-metrics-tls\") pod \"dns-default-gvn74\" (UID: \"6a046e7f-51bb-42f3-be5a-e3f15a80afce\") " pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:35.693350 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.693329 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls\") pod \"image-registry-7774cfc878-x5gcd\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:35.842959 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.842930 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-59nw8\"" Sep 19 10:35:35.847398 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.847380 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:35.849977 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.849959 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-snddw\"" Sep 19 10:35:35.857778 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.857760 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:35.892459 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.892425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:35:35.892568 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.892554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs\") pod \"network-metrics-daemon-9xvrh\" (UID: \"1728a47c-9260-477e-8b40-42c3cfdee6db\") " pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:35:35.895773 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.895753 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 19 10:35:35.895860 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.895753 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 19 10:35:35.905047 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.905001 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4345b3c-abd0-4cc2-8b21-a87b10324f78-original-pull-secret\") pod \"global-pull-secret-syncer-d2djw\" (UID: \"f4345b3c-abd0-4cc2-8b21-a87b10324f78\") " pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:35:35.905162 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.905090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1728a47c-9260-477e-8b40-42c3cfdee6db-metrics-certs\") pod \"network-metrics-daemon-9xvrh\" (UID: \"1728a47c-9260-477e-8b40-42c3cfdee6db\") " pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:35:35.993639 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.993604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npj58\" (UniqueName: \"kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58\") pod \"network-check-target-ntfqc\" (UID: \"65cc3016-6e8a-4a8c-be7b-3634a31508a1\") " pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:35:35.995905 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:35.995887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npj58\" (UniqueName: \"kubernetes.io/projected/65cc3016-6e8a-4a8c-be7b-3634a31508a1-kube-api-access-npj58\") pod \"network-check-target-ntfqc\" (UID: \"65cc3016-6e8a-4a8c-be7b-3634a31508a1\") " pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:35:36.099405 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.099381 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2djw" Sep 19 10:35:36.108369 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.108352 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d9kv4\"" Sep 19 10:35:36.111615 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.111601 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8jxmp\"" Sep 19 10:35:36.115762 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.115746 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:35:36.119958 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.119939 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xvrh" Sep 19 10:35:36.156114 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.156093 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7774cfc878-x5gcd"] Sep 19 10:35:36.160001 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:36.159978 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a1b6fa_ab80_4027_981f_5daf44a44b61.slice/crio-2e0f8c6b3d591b0c788ee7c8a0e765fe78141228310fcbb555eb2759850b8731 WatchSource:0}: Error finding container 2e0f8c6b3d591b0c788ee7c8a0e765fe78141228310fcbb555eb2759850b8731: Status 404 returned error can't find the container with id 2e0f8c6b3d591b0c788ee7c8a0e765fe78141228310fcbb555eb2759850b8731 Sep 19 10:35:36.183745 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.182815 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gvn74"] Sep 19 10:35:36.190371 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:36.190342 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a046e7f_51bb_42f3_be5a_e3f15a80afce.slice/crio-13d9a83588bb8bc611d376886dbda0a0f9ab9b2690851f12a9dcf4cf0a42e9dc WatchSource:0}: Error finding container 13d9a83588bb8bc611d376886dbda0a0f9ab9b2690851f12a9dcf4cf0a42e9dc: Status 404 returned error can't find the container with id 13d9a83588bb8bc611d376886dbda0a0f9ab9b2690851f12a9dcf4cf0a42e9dc Sep 19 10:35:36.400367 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.400306 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d2djw"] Sep 19 10:35:36.403088 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:36.403046 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4345b3c_abd0_4cc2_8b21_a87b10324f78.slice/crio-7a10222549c37a1dac7ef418b5f1807b5ec653dfec29ca1e55acfedb44e85456 WatchSource:0}: Error finding container 7a10222549c37a1dac7ef418b5f1807b5ec653dfec29ca1e55acfedb44e85456: Status 404 returned error can't find the container with id 7a10222549c37a1dac7ef418b5f1807b5ec653dfec29ca1e55acfedb44e85456 Sep 19 10:35:36.429536 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.429513 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ntfqc"] Sep 19 10:35:36.431277 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:36.431255 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65cc3016_6e8a_4a8c_be7b_3634a31508a1.slice/crio-7c326bd194a87020dfa70c95b87f4ab04efb461757562f78f22b59c7616b87bc WatchSource:0}: Error finding container 7c326bd194a87020dfa70c95b87f4ab04efb461757562f78f22b59c7616b87bc: Status 404 returned error can't find the container with id 7c326bd194a87020dfa70c95b87f4ab04efb461757562f78f22b59c7616b87bc Sep 19 10:35:36.469575 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.469552 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9xvrh"] Sep 19 10:35:36.473043 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:36.473021 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1728a47c_9260_477e_8b40_42c3cfdee6db.slice/crio-5d1f7fed4f260bbefc20872110eace30a389f985e059bd18668053b816620094 WatchSource:0}: Error finding container 5d1f7fed4f260bbefc20872110eace30a389f985e059bd18668053b816620094: Status 404 returned error can't find the container with id 5d1f7fed4f260bbefc20872110eace30a389f985e059bd18668053b816620094 Sep 19 10:35:36.630618 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.630586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9xvrh" event={"ID":"1728a47c-9260-477e-8b40-42c3cfdee6db","Type":"ContainerStarted","Data":"5d1f7fed4f260bbefc20872110eace30a389f985e059bd18668053b816620094"} Sep 19 10:35:36.632197 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.632170 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ntfqc" event={"ID":"65cc3016-6e8a-4a8c-be7b-3634a31508a1","Type":"ContainerStarted","Data":"540b6642b4547014b891259efc2a7f8da13f3dc185b830ff19ee26ee0af07cdb"} Sep 19 10:35:36.632332 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.632216 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ntfqc" event={"ID":"65cc3016-6e8a-4a8c-be7b-3634a31508a1","Type":"ContainerStarted","Data":"7c326bd194a87020dfa70c95b87f4ab04efb461757562f78f22b59c7616b87bc"} Sep 19 10:35:36.632332 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.632240 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:35:36.633382 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.633361 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gvn74" event={"ID":"6a046e7f-51bb-42f3-be5a-e3f15a80afce","Type":"ContainerStarted","Data":"13d9a83588bb8bc611d376886dbda0a0f9ab9b2690851f12a9dcf4cf0a42e9dc"} Sep 19 10:35:36.634976 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.634951 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" event={"ID":"13a1b6fa-ab80-4027-981f-5daf44a44b61","Type":"ContainerStarted","Data":"c86b4aa01dfeda78b8e71e6545f3c7f9ea8b7acaa64d0c3e244fa00284b5ca95"} Sep 19 10:35:36.635092 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.634997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" event={"ID":"13a1b6fa-ab80-4027-981f-5daf44a44b61","Type":"ContainerStarted","Data":"2e0f8c6b3d591b0c788ee7c8a0e765fe78141228310fcbb555eb2759850b8731"} Sep 19 10:35:36.635092 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.635024 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:36.636342 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.636314 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d2djw" event={"ID":"f4345b3c-abd0-4cc2-8b21-a87b10324f78","Type":"ContainerStarted","Data":"7a10222549c37a1dac7ef418b5f1807b5ec653dfec29ca1e55acfedb44e85456"} Sep 19 10:35:36.649568 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.649518 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ntfqc" podStartSLOduration=66.649503192 podStartE2EDuration="1m6.649503192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:36.648492507 +0000 UTC m=+68.560997224" watchObservedRunningTime="2026-09-19 10:35:36.649503192 +0000 UTC m=+68.562007899" Sep 19 10:35:36.669106 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:36.669017 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" podStartSLOduration=66.669003734 podStartE2EDuration="1m6.669003734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:36.668402836 +0000 UTC m=+68.580907553" watchObservedRunningTime="2026-09-19 10:35:36.669003734 +0000 UTC m=+68.581508450" Sep 19 10:35:38.423491 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:38.423447 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:38.423887 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:38.423518 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:35:38.423887 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:38.423864 2579 scope.go:117] "RemoveContainer" containerID="318dc4f2782bcb9182240b9ebeabaa3ea56f4ab193f6ae8e456cd099aa77a0fa" Sep 19 10:35:38.424052 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:38.424030 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-xnm4h_openshift-console-operator(61826a91-39c7-4d1b-a0f9-c1de1de49c0e)\"" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" podUID="61826a91-39c7-4d1b-a0f9-c1de1de49c0e" Sep 19 10:35:39.651094 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:39.651038 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9xvrh" event={"ID":"1728a47c-9260-477e-8b40-42c3cfdee6db","Type":"ContainerStarted","Data":"7235467e14b263a82b01edb2f431106f488c999e590f65a7fcb88318a6b3cf65"} Sep 19 10:35:39.651573 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:39.651155 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9xvrh" event={"ID":"1728a47c-9260-477e-8b40-42c3cfdee6db","Type":"ContainerStarted","Data":"7b2fbeb858486be0e1da499f2b9139adfdc72d0c59486d9b2baaff316055504e"} Sep 19 10:35:39.653164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:39.653136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gvn74" event={"ID":"6a046e7f-51bb-42f3-be5a-e3f15a80afce","Type":"ContainerStarted","Data":"2ba7a861b31542d7ee548f56c6b26f76ed9d56c8d32327282f86f602eb39af57"} Sep 19 10:35:39.653283 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:39.653188 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gvn74" event={"ID":"6a046e7f-51bb-42f3-be5a-e3f15a80afce","Type":"ContainerStarted","Data":"69310ffad56d0de111dc0e0a532ef68f3a55143c64002db180e15f5f43760ad9"} Sep 19 10:35:39.653330 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:39.653280 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:39.667964 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:39.667923 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9xvrh" podStartSLOduration=67.545270233 podStartE2EDuration="1m9.667908944s" totalImagesPullingTime="2.122638711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:39.666873951 +0000 UTC m=+71.579378669" watchObservedRunningTime="2026-09-19 10:35:39.667908944 +0000 UTC m=+71.580413659" Sep 19 10:35:39.685885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:39.685838 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gvn74" podStartSLOduration=34.277729442 podStartE2EDuration="36.685823264s" totalImagesPullingTime="2.408093822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:39.684808821 +0000 UTC m=+71.597313548" watchObservedRunningTime="2026-09-19 10:35:39.685823264 +0000 UTC m=+71.598328019" Sep 19 10:35:40.131369 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.131288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:40.131527 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.131384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jsd8\" (UID: \"e1e97edd-62b0-443a-b1cc-d7b796704644\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:40.134141 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.134109 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/245f29a4-837b-4197-9eb7-52f324d83901-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-9rtmm\" (UID: \"245f29a4-837b-4197-9eb7-52f324d83901\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:40.134252 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.134115 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1e97edd-62b0-443a-b1cc-d7b796704644-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jsd8\" (UID: \"e1e97edd-62b0-443a-b1cc-d7b796704644\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:40.232279 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.232250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:40.232463 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.232311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:40.232996 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.232962 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b92f4b1-f19a-42ac-b87b-835f23d214cd-service-ca-bundle\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:40.234977 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.234952 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b92f4b1-f19a-42ac-b87b-835f23d214cd-metrics-certs\") pod \"router-default-9998785cb-l2lkw\" (UID: \"6b92f4b1-f19a-42ac-b87b-835f23d214cd\") " pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:40.293997 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.293973 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-54v4j\"" Sep 19 10:35:40.302180 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.302158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" Sep 19 10:35:40.339439 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.339418 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qkdwx\"" Sep 19 10:35:40.346724 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.346706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" Sep 19 10:35:40.362119 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.362097 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5b9hv\"" Sep 19 10:35:40.370351 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.370328 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:40.843886 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.843794 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm"] Sep 19 10:35:40.847500 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:40.847464 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245f29a4_837b_4197_9eb7_52f324d83901.slice/crio-2f83e41ce09f93c6a1d47cf4edad56216f1bf1de4801e9a312c08ddf97721b10 WatchSource:0}: Error finding container 2f83e41ce09f93c6a1d47cf4edad56216f1bf1de4801e9a312c08ddf97721b10: Status 404 returned error can't find the container with id 2f83e41ce09f93c6a1d47cf4edad56216f1bf1de4801e9a312c08ddf97721b10 Sep 19 10:35:40.849229 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.849206 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8"] Sep 19 10:35:40.852491 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:40.852469 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9998785cb-l2lkw"] Sep 19 10:35:40.852712 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:40.852694 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e97edd_62b0_443a_b1cc_d7b796704644.slice/crio-8f94a869a082070118cdb5e84a22feefea7c301b0263461a4bb6b7559ac717a6 WatchSource:0}: Error finding container 8f94a869a082070118cdb5e84a22feefea7c301b0263461a4bb6b7559ac717a6: Status 404 returned error can't find the container with id 8f94a869a082070118cdb5e84a22feefea7c301b0263461a4bb6b7559ac717a6 Sep 19 10:35:40.855145 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:40.855108 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b92f4b1_f19a_42ac_b87b_835f23d214cd.slice/crio-aca240a9c8788c26e2a3050340cd1789271c6e006babc8696ac997ef7130b484 WatchSource:0}: Error finding container aca240a9c8788c26e2a3050340cd1789271c6e006babc8696ac997ef7130b484: Status 404 returned error can't find the container with id aca240a9c8788c26e2a3050340cd1789271c6e006babc8696ac997ef7130b484 Sep 19 10:35:41.664601 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:41.664557 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" event={"ID":"245f29a4-837b-4197-9eb7-52f324d83901","Type":"ContainerStarted","Data":"2f83e41ce09f93c6a1d47cf4edad56216f1bf1de4801e9a312c08ddf97721b10"} Sep 19 10:35:41.665961 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:41.665935 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d2djw" event={"ID":"f4345b3c-abd0-4cc2-8b21-a87b10324f78","Type":"ContainerStarted","Data":"9a1c0a061443b9c453bd3883c8538ea1cfe36783893eb9f93eb4d45bc19eef82"} Sep 19 10:35:41.667518 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:41.667497 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9998785cb-l2lkw" event={"ID":"6b92f4b1-f19a-42ac-b87b-835f23d214cd","Type":"ContainerStarted","Data":"12b348c09e7c0308fa62ec7363869e1e661d2d4388b6a09ba0bc28a44370d313"} Sep 19 10:35:41.667605 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:41.667538 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9998785cb-l2lkw" event={"ID":"6b92f4b1-f19a-42ac-b87b-835f23d214cd","Type":"ContainerStarted","Data":"aca240a9c8788c26e2a3050340cd1789271c6e006babc8696ac997ef7130b484"} Sep 19 10:35:41.668596 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:41.668574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" event={"ID":"e1e97edd-62b0-443a-b1cc-d7b796704644","Type":"ContainerStarted","Data":"8f94a869a082070118cdb5e84a22feefea7c301b0263461a4bb6b7559ac717a6"} Sep 19 10:35:41.682465 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:41.682411 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d2djw" podStartSLOduration=67.471913485 podStartE2EDuration="1m11.682394665s" totalImagesPullingTime="4.21048118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 10:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:41.681802626 +0000 UTC m=+73.594307343" watchObservedRunningTime="2026-09-19 10:35:41.682394665 +0000 UTC m=+73.594899382" Sep 19 10:35:41.701797 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:41.701715 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-9998785cb-l2lkw" podStartSLOduration=33.701701312 podStartE2EDuration="33.701701312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:41.700042132 +0000 UTC m=+73.612546850" watchObservedRunningTime="2026-09-19 10:35:41.701701312 +0000 UTC m=+73.614206026" Sep 19 10:35:42.371251 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:42.371208 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:42.374220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:42.374196 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:42.674393 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:42.674357 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" event={"ID":"245f29a4-837b-4197-9eb7-52f324d83901","Type":"ContainerStarted","Data":"e7cd12826373ac448457827352cea9102d71105366eb75af1c64fdfdf2168986"} Sep 19 10:35:42.676295 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:42.675944 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" event={"ID":"e1e97edd-62b0-443a-b1cc-d7b796704644","Type":"ContainerStarted","Data":"e79f1697a948bffe1301c6a39a61a7c0fc9be4f12a39dc22f9e1baec5c526a59"} Sep 19 10:35:42.676428 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:42.676401 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:42.677721 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:42.677702 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-9998785cb-l2lkw" Sep 19 10:35:42.691868 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:42.691821 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-9rtmm" podStartSLOduration=32.969318905 podStartE2EDuration="34.691807545s" totalImagesPullingTime="1.72248864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:42.69141377 +0000 UTC m=+74.603918486" watchObservedRunningTime="2026-09-19 10:35:42.691807545 +0000 UTC m=+74.604312261" Sep 19 10:35:42.726938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:42.726872 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jsd8" podStartSLOduration=33.011605417 podStartE2EDuration="34.726849186s" totalImagesPullingTime="1.715243769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:42.725844985 +0000 UTC m=+74.638349705" watchObservedRunningTime="2026-09-19 10:35:42.726849186 +0000 UTC m=+74.639353985" Sep 19 10:35:43.077303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:43.077200 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn"] Sep 19 10:35:43.079257 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:43.079239 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn" Sep 19 10:35:43.081923 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:43.081898 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 19 10:35:43.082023 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:43.081932 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vvq86\"" Sep 19 10:35:43.088306 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:43.088277 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn"] Sep 19 10:35:43.257855 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:43.257828 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d6b50e84-4a6f-469c-bbab-e8524c838662-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-f6lqn\" (UID: \"d6b50e84-4a6f-469c-bbab-e8524c838662\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn" Sep 19 10:35:43.358180 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:43.358120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d6b50e84-4a6f-469c-bbab-e8524c838662-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-f6lqn\" (UID: \"d6b50e84-4a6f-469c-bbab-e8524c838662\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn" Sep 19 10:35:43.358275 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:43.358233 2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 19 10:35:43.358320 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:43.358286 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6b50e84-4a6f-469c-bbab-e8524c838662-tls-certificates podName:d6b50e84-4a6f-469c-bbab-e8524c838662 nodeName:}" failed. No retries permitted until 2026-09-19 10:35:43.858270281 +0000 UTC m=+75.770774974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/d6b50e84-4a6f-469c-bbab-e8524c838662-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-f6lqn" (UID: "d6b50e84-4a6f-469c-bbab-e8524c838662") : secret "prometheus-operator-admission-webhook-tls" not found Sep 19 10:35:43.862011 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:43.861972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d6b50e84-4a6f-469c-bbab-e8524c838662-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-f6lqn\" (UID: \"d6b50e84-4a6f-469c-bbab-e8524c838662\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn" Sep 19 10:35:43.864747 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:43.864718 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d6b50e84-4a6f-469c-bbab-e8524c838662-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-f6lqn\" (UID: \"d6b50e84-4a6f-469c-bbab-e8524c838662\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn" Sep 19 10:35:43.988318 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:43.988292 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn" Sep 19 10:35:44.254689 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:44.254661 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn"] Sep 19 10:35:44.257387 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:44.257364 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b50e84_4a6f_469c_bbab_e8524c838662.slice/crio-c860a0e9aed86db6ca1dbfb5121a42b5df547493133e3e8b196aebcf5c9a5522 WatchSource:0}: Error finding container c860a0e9aed86db6ca1dbfb5121a42b5df547493133e3e8b196aebcf5c9a5522: Status 404 returned error can't find the container with id c860a0e9aed86db6ca1dbfb5121a42b5df547493133e3e8b196aebcf5c9a5522 Sep 19 10:35:44.683607 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:44.683573 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn" event={"ID":"d6b50e84-4a6f-469c-bbab-e8524c838662","Type":"ContainerStarted","Data":"c860a0e9aed86db6ca1dbfb5121a42b5df547493133e3e8b196aebcf5c9a5522"} Sep 19 10:35:45.688641 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:45.688605 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn" event={"ID":"d6b50e84-4a6f-469c-bbab-e8524c838662","Type":"ContainerStarted","Data":"6ecf5033cedb267fbcc959ee505d774f84b44dde4722502ddd55ce1b8ec99bde"} Sep 19 10:35:45.689082 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:45.688810 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn" Sep 19 10:35:45.693500 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:45.693477 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn" Sep 19 10:35:45.707682 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:45.707635 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-f6lqn" podStartSLOduration=1.596494971 podStartE2EDuration="2.707619224s" totalImagesPullingTime="1.111124253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:45.70629098 +0000 UTC m=+77.618795696" watchObservedRunningTime="2026-09-19 10:35:45.707619224 +0000 UTC m=+77.620123940" Sep 19 10:35:46.140523 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.140494 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c"] Sep 19 10:35:46.142702 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.142685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.146034 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.146014 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 19 10:35:46.146264 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.146247 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 19 10:35:46.147496 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.147475 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-c7kc9\"" Sep 19 10:35:46.147644 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.147508 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 19 10:35:46.156904 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.156885 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c"] Sep 19 10:35:46.279013 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.278981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-xxr9c\" (UID: \"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.279138 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.279020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb822\" (UniqueName: \"kubernetes.io/projected/49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1-kube-api-access-tb822\") pod \"prometheus-operator-7bd5d56d6-xxr9c\" (UID: \"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.279138 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.279086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-xxr9c\" (UID: \"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.279138 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.279111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-xxr9c\" (UID: \"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.379485 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.379453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-xxr9c\" (UID: \"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.379485 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.379485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-xxr9c\" (UID: \"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.379619 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.379516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-xxr9c\" (UID: \"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.379619 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.379549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb822\" (UniqueName: \"kubernetes.io/projected/49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1-kube-api-access-tb822\") pod \"prometheus-operator-7bd5d56d6-xxr9c\" (UID: \"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.380219 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.380198 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-xxr9c\" (UID: \"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.381773 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.381744 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-xxr9c\" (UID: \"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.381877 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.381859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-xxr9c\" (UID: \"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.388583 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.388563 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb822\" (UniqueName: \"kubernetes.io/projected/49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1-kube-api-access-tb822\") pod \"prometheus-operator-7bd5d56d6-xxr9c\" (UID: \"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.451492 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.451432 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" Sep 19 10:35:46.715970 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:46.715893 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c"] Sep 19 10:35:46.718673 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:46.718642 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e6bdc8_aa3e_4432_8ae1_f35708c6e8d1.slice/crio-3ddbf03e67469323c802649afc972aea7dcfe8116b02b1c7640147a2bf1f2a19 WatchSource:0}: Error finding container 3ddbf03e67469323c802649afc972aea7dcfe8116b02b1c7640147a2bf1f2a19: Status 404 returned error can't find the container with id 3ddbf03e67469323c802649afc972aea7dcfe8116b02b1c7640147a2bf1f2a19 Sep 19 10:35:47.697470 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:47.697436 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" event={"ID":"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1","Type":"ContainerStarted","Data":"3ddbf03e67469323c802649afc972aea7dcfe8116b02b1c7640147a2bf1f2a19"} Sep 19 10:35:48.703020 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:48.702982 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" event={"ID":"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1","Type":"ContainerStarted","Data":"179cd503be4ab357fd3be18e4eefe9c5c815b45addee133ad9e4e77516ccce23"} Sep 19 10:35:48.703390 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:48.703034 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" event={"ID":"49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1","Type":"ContainerStarted","Data":"f02d44fa977d75a4e955dd42d6c94b0eb7f7c8d1331f8a7ddef77c014708c63c"} Sep 19 10:35:48.725014 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:48.724964 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xxr9c" podStartSLOduration=1.465210452 podStartE2EDuration="2.724950891s" totalImagesPullingTime="1.259740439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:48.720350854 +0000 UTC m=+80.632855570" watchObservedRunningTime="2026-09-19 10:35:48.724950891 +0000 UTC m=+80.637455607" Sep 19 10:35:49.662771 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:49.662744 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gvn74" Sep 19 10:35:50.191846 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.191813 2579 scope.go:117] "RemoveContainer" containerID="318dc4f2782bcb9182240b9ebeabaa3ea56f4ab193f6ae8e456cd099aa77a0fa" Sep 19 10:35:50.192297 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:50.192002 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-xnm4h_openshift-console-operator(61826a91-39c7-4d1b-a0f9-c1de1de49c0e)\"" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" podUID="61826a91-39c7-4d1b-a0f9-c1de1de49c0e" Sep 19 10:35:50.634742 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.634712 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-wzdmh"] Sep 19 10:35:50.636976 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.636956 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.639568 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.639548 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-g5zfc\"" Sep 19 10:35:50.639817 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.639802 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 19 10:35:50.640046 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.640025 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 19 10:35:50.640728 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.640713 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 19 10:35:50.649802 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.649784 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bkpcd"] Sep 19 10:35:50.651763 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.651749 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.654325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.654308 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 19 10:35:50.654394 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.654380 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 19 10:35:50.654462 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.654443 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 19 10:35:50.654521 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.654483 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zsbvs\"" Sep 19 10:35:50.663518 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.663497 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-wzdmh"] Sep 19 10:35:50.706316 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706286 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75607ff6-c200-4323-9865-263ec4057fef-sys\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.706445 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-wtmp\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.706445 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/72a1da39-f5b2-46e2-a508-9d3b71eed28e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.706572 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706462 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75607ff6-c200-4323-9865-263ec4057fef-metrics-client-ca\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.706572 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72a1da39-f5b2-46e2-a508-9d3b71eed28e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.706572 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckhz9\" (UniqueName: \"kubernetes.io/projected/72a1da39-f5b2-46e2-a508-9d3b71eed28e-kube-api-access-ckhz9\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.706572 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-tls\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.706764 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72a1da39-f5b2-46e2-a508-9d3b71eed28e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.706764 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/75607ff6-c200-4323-9865-263ec4057fef-root\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.706764 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.706764 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/72a1da39-f5b2-46e2-a508-9d3b71eed28e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.706764 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62vc\" (UniqueName: \"kubernetes.io/projected/75607ff6-c200-4323-9865-263ec4057fef-kube-api-access-s62vc\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.706764 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-textfile\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.706764 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-accelerators-collector-config\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.707106 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.706784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/72a1da39-f5b2-46e2-a508-9d3b71eed28e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.807903 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.807866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.808033 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.807909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/72a1da39-f5b2-46e2-a508-9d3b71eed28e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.808033 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.807927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62vc\" (UniqueName: \"kubernetes.io/projected/75607ff6-c200-4323-9865-263ec4057fef-kube-api-access-s62vc\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.808033 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.807944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-textfile\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.808218 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808105 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-accelerators-collector-config\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.808218 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808157 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/72a1da39-f5b2-46e2-a508-9d3b71eed28e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.808319 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75607ff6-c200-4323-9865-263ec4057fef-sys\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.808319 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808296 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75607ff6-c200-4323-9865-263ec4057fef-sys\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.808319 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808304 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-textfile\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.808468 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-wtmp\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.808468 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/72a1da39-f5b2-46e2-a508-9d3b71eed28e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.808468 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75607ff6-c200-4323-9865-263ec4057fef-metrics-client-ca\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.808468 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808452 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72a1da39-f5b2-46e2-a508-9d3b71eed28e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.808664 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckhz9\" (UniqueName: \"kubernetes.io/projected/72a1da39-f5b2-46e2-a508-9d3b71eed28e-kube-api-access-ckhz9\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.808664 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808535 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-wtmp\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.808664 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808556 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/72a1da39-f5b2-46e2-a508-9d3b71eed28e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.808664 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-tls\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.808858 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72a1da39-f5b2-46e2-a508-9d3b71eed28e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.808858 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/75607ff6-c200-4323-9865-263ec4057fef-root\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.808858 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:50.808773 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 19 10:35:50.808858 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.808805 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/75607ff6-c200-4323-9865-263ec4057fef-root\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.808858 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:35:50.808839 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-tls podName:75607ff6-c200-4323-9865-263ec4057fef nodeName:}" failed. No retries permitted until 2026-09-19 10:35:51.308819707 +0000 UTC m=+83.221324412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-tls") pod "node-exporter-bkpcd" (UID: "75607ff6-c200-4323-9865-263ec4057fef") : secret "node-exporter-tls" not found Sep 19 10:35:50.809510 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.809462 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/72a1da39-f5b2-46e2-a508-9d3b71eed28e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.809702 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.809678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72a1da39-f5b2-46e2-a508-9d3b71eed28e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.809965 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.809943 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-accelerators-collector-config\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.810384 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.810359 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75607ff6-c200-4323-9865-263ec4057fef-metrics-client-ca\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.810794 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.810775 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/72a1da39-f5b2-46e2-a508-9d3b71eed28e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.811125 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.811104 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.811447 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.811431 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72a1da39-f5b2-46e2-a508-9d3b71eed28e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.816578 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.816555 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckhz9\" (UniqueName: \"kubernetes.io/projected/72a1da39-f5b2-46e2-a508-9d3b71eed28e-kube-api-access-ckhz9\") pod \"kube-state-metrics-84c885658c-wzdmh\" (UID: \"72a1da39-f5b2-46e2-a508-9d3b71eed28e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:50.817126 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.817108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62vc\" (UniqueName: \"kubernetes.io/projected/75607ff6-c200-4323-9865-263ec4057fef-kube-api-access-s62vc\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:50.946478 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:50.946423 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" Sep 19 10:35:51.217525 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:51.217422 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-wzdmh"] Sep 19 10:35:51.220006 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:51.219980 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a1da39_f5b2_46e2_a508_9d3b71eed28e.slice/crio-aa09f3dc4a171d31cf05fe09d28e6d0387ad4ef9d6438eb3725b9ef025f0601f WatchSource:0}: Error finding container aa09f3dc4a171d31cf05fe09d28e6d0387ad4ef9d6438eb3725b9ef025f0601f: Status 404 returned error can't find the container with id aa09f3dc4a171d31cf05fe09d28e6d0387ad4ef9d6438eb3725b9ef025f0601f Sep 19 10:35:51.313710 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:51.313680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-tls\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:51.315812 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:51.315789 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/75607ff6-c200-4323-9865-263ec4057fef-node-exporter-tls\") pod \"node-exporter-bkpcd\" (UID: \"75607ff6-c200-4323-9865-263ec4057fef\") " pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:51.560490 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:51.560422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bkpcd" Sep 19 10:35:51.571326 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:51.571293 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75607ff6_c200_4323_9865_263ec4057fef.slice/crio-0e0d141db4858b7f47ac917fd263b661e88cd29f7a24337f9ab4257a49d35fa7 WatchSource:0}: Error finding container 0e0d141db4858b7f47ac917fd263b661e88cd29f7a24337f9ab4257a49d35fa7: Status 404 returned error can't find the container with id 0e0d141db4858b7f47ac917fd263b661e88cd29f7a24337f9ab4257a49d35fa7 Sep 19 10:35:51.721710 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:51.721673 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" event={"ID":"72a1da39-f5b2-46e2-a508-9d3b71eed28e","Type":"ContainerStarted","Data":"aa09f3dc4a171d31cf05fe09d28e6d0387ad4ef9d6438eb3725b9ef025f0601f"} Sep 19 10:35:51.723143 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:51.723102 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bkpcd" event={"ID":"75607ff6-c200-4323-9865-263ec4057fef","Type":"ContainerStarted","Data":"0e0d141db4858b7f47ac917fd263b661e88cd29f7a24337f9ab4257a49d35fa7"} Sep 19 10:35:52.729840 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:52.729806 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" event={"ID":"72a1da39-f5b2-46e2-a508-9d3b71eed28e","Type":"ContainerStarted","Data":"9bb5d43461b3ce82cb10d641a05defd50fd1ae52a29d9ea0e31b3be45e46d6d4"} Sep 19 10:35:52.732036 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:52.732007 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bkpcd" event={"ID":"75607ff6-c200-4323-9865-263ec4057fef","Type":"ContainerStarted","Data":"0ceb0660fec7986694953fe00f911969778d0d2da10a73e7ed4fce5901b18e20"} Sep 19 10:35:53.738718 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:53.738688 2579 generic.go:356] "Generic (PLEG): container finished" podID="75607ff6-c200-4323-9865-263ec4057fef" containerID="0ceb0660fec7986694953fe00f911969778d0d2da10a73e7ed4fce5901b18e20" exitCode=0 Sep 19 10:35:53.739144 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:53.738739 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bkpcd" event={"ID":"75607ff6-c200-4323-9865-263ec4057fef","Type":"ContainerDied","Data":"0ceb0660fec7986694953fe00f911969778d0d2da10a73e7ed4fce5901b18e20"} Sep 19 10:35:53.741209 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:53.741180 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" event={"ID":"72a1da39-f5b2-46e2-a508-9d3b71eed28e","Type":"ContainerStarted","Data":"89b06ccf33af26a88307d544c09cc0cd1c203bbdfd2d1a1825dc096cabcccf64"} Sep 19 10:35:53.741327 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:53.741236 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" event={"ID":"72a1da39-f5b2-46e2-a508-9d3b71eed28e","Type":"ContainerStarted","Data":"952e176e93810186c45cb191b5b4efe89706f5c30374e943ea381c4ce2817be9"} Sep 19 10:35:53.826279 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:53.826236 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-wzdmh" podStartSLOduration=2.428783838 podStartE2EDuration="3.826219854s" totalImagesPullingTime="1.397436016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:53.822958418 +0000 UTC m=+85.735463135" watchObservedRunningTime="2026-09-19 10:35:53.826219854 +0000 UTC m=+85.738724570" Sep 19 10:35:54.747588 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:54.747544 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bkpcd" event={"ID":"75607ff6-c200-4323-9865-263ec4057fef","Type":"ContainerStarted","Data":"63bcc680f899634536b264ba01ccdaf960098b6546993758a949389fc68c702c"} Sep 19 10:35:54.748100 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:54.747636 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bkpcd" event={"ID":"75607ff6-c200-4323-9865-263ec4057fef","Type":"ContainerStarted","Data":"882c336019fc435370c9b8969388e1b24ee2fb6aa9a2795f3137a6a1c20a0c9d"} Sep 19 10:35:54.767924 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:54.767878 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bkpcd" podStartSLOduration=3.627451786 podStartE2EDuration="4.767863612s" totalImagesPullingTime="1.047598197s" totalInitContainerRuntime="92.813629ms" isStatefulPod=true podCreationTimestamp="2026-09-19 10:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:35:54.766327921 +0000 UTC m=+86.678832650" watchObservedRunningTime="2026-09-19 10:35:54.767863612 +0000 UTC m=+86.680368328" Sep 19 10:35:56.777351 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.777317 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 19 10:35:56.780595 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.780573 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.783116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.783078 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 19 10:35:56.784118 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.784099 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 19 10:35:56.784202 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.784151 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 19 10:35:56.785197 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.785181 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 19 10:35:56.785395 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.785357 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 19 10:35:56.786902 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.786887 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 19 10:35:56.787178 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.787112 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 19 10:35:56.787178 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.787142 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 19 10:35:56.790489 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.788260 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 19 10:35:56.790489 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.788442 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hvv6k\"" Sep 19 10:35:56.790489 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.789379 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 19 10:35:56.790489 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.789673 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 19 10:35:56.790489 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.789843 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-42dugplt7ovcs\"" Sep 19 10:35:56.790489 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.790024 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 19 10:35:56.791158 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.790633 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 19 10:35:56.792050 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.792030 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 19 10:35:56.794167 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.794148 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 19 10:35:56.805509 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.805425 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 19 10:35:56.860684 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.860647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.860841 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.860690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0914fa69-fbd9-4be4-8189-95708c8ad9c6-config-out\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.860841 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.860713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.860841 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.860734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.860841 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.860752 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.860841 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.860778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.860862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0914fa69-fbd9-4be4-8189-95708c8ad9c6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.860898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.860925 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.860957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.860992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-config\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.861016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.861052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.861090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4clfz\" (UniqueName: \"kubernetes.io/projected/0914fa69-fbd9-4be4-8189-95708c8ad9c6-kube-api-access-4clfz\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861443 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.861120 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861443 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.861146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861443 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.861171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861443 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.861198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861443 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.861271 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.861443 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.861295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-web-config\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962397 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962397 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962610 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962610 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962610 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962610 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-web-config\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962610 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962610 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0914fa69-fbd9-4be4-8189-95708c8ad9c6-config-out\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962610 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0914fa69-fbd9-4be4-8189-95708c8ad9c6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-config\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.962938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.962891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4clfz\" (UniqueName: \"kubernetes.io/projected/0914fa69-fbd9-4be4-8189-95708c8ad9c6-kube-api-access-4clfz\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.963468 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.963176 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.964154 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.963832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.967560 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.967375 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.968220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.967629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.968220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.967894 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-web-config\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.968220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.967931 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.968220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.968005 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.968433 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.968289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.968639 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.968615 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.969485 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.969290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.970366 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.970346 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.977306 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.970966 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0914fa69-fbd9-4be4-8189-95708c8ad9c6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.977306 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.971009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.977306 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.971246 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.977306 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.971303 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-config\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.977306 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.971645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.977306 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.971818 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0914fa69-fbd9-4be4-8189-95708c8ad9c6-config-out\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.977306 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.972640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.977766 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.977740 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4clfz\" (UniqueName: \"kubernetes.io/projected/0914fa69-fbd9-4be4-8189-95708c8ad9c6-kube-api-access-4clfz\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:56.977869 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:56.977777 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:57.092306 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:57.092223 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:35:57.386071 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:57.386034 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 19 10:35:57.387895 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:35:57.387866 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0914fa69_fbd9_4be4_8189_95708c8ad9c6.slice/crio-e4a4c745d6307ed72b65bb00bb8a52beea880f8880497eb48668b8e0c8cfd2dc WatchSource:0}: Error finding container e4a4c745d6307ed72b65bb00bb8a52beea880f8880497eb48668b8e0c8cfd2dc: Status 404 returned error can't find the container with id e4a4c745d6307ed72b65bb00bb8a52beea880f8880497eb48668b8e0c8cfd2dc Sep 19 10:35:57.645737 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:57.645663 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:35:57.762930 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:57.762894 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerStarted","Data":"e4a4c745d6307ed72b65bb00bb8a52beea880f8880497eb48668b8e0c8cfd2dc"} Sep 19 10:35:58.768316 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:58.768271 2579 generic.go:356] "Generic (PLEG): container finished" podID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerID="14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf" exitCode=0 Sep 19 10:35:58.768758 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:58.768366 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerDied","Data":"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf"} Sep 19 10:35:59.412937 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:35:59.412899 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7774cfc878-x5gcd"] Sep 19 10:36:02.790013 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:02.789974 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerStarted","Data":"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765"} Sep 19 10:36:02.790410 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:02.790046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerStarted","Data":"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb"} Sep 19 10:36:03.798638 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:03.798600 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerStarted","Data":"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911"} Sep 19 10:36:03.798950 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:03.798697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerStarted","Data":"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87"} Sep 19 10:36:03.798950 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:03.798725 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerStarted","Data":"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b"} Sep 19 10:36:03.798950 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:03.798753 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerStarted","Data":"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128"} Sep 19 10:36:03.829355 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:03.829302 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.312072354 podStartE2EDuration="7.829046767s" totalImagesPullingTime="5.499908666s" totalInitContainerRuntime="17.065747ms" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:03.827259501 +0000 UTC m=+95.739764232" watchObservedRunningTime="2026-09-19 10:36:03.829046767 +0000 UTC m=+95.741551483" Sep 19 10:36:04.190369 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:04.190336 2579 scope.go:117] "RemoveContainer" containerID="318dc4f2782bcb9182240b9ebeabaa3ea56f4ab193f6ae8e456cd099aa77a0fa" Sep 19 10:36:04.805386 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:04.805355 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 10:36:04.806661 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:04.806635 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" event={"ID":"61826a91-39c7-4d1b-a0f9-c1de1de49c0e","Type":"ContainerStarted","Data":"d7fe24347024bd0df57ff7810a83264503bad9bc108a909716bff3121fa7e018"} Sep 19 10:36:04.807123 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:04.807069 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:36:04.837953 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:04.837906 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" podStartSLOduration=51.794344347 podStartE2EDuration="56.837892314s" totalImagesPullingTime="5.043547967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:36:04.836221234 +0000 UTC m=+96.748725985" watchObservedRunningTime="2026-09-19 10:36:04.837892314 +0000 UTC m=+96.750397029" Sep 19 10:36:05.348525 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:05.348492 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-xnm4h" Sep 19 10:36:07.092709 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:07.092665 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:36:07.643582 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:07.643541 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ntfqc" Sep 19 10:36:24.436866 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.436796 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" podUID="13a1b6fa-ab80-4027-981f-5daf44a44b61" containerName="registry" containerID="cri-o://c86b4aa01dfeda78b8e71e6545f3c7f9ea8b7acaa64d0c3e244fa00284b5ca95" gracePeriod=30 Sep 19 10:36:24.673477 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.673453 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:36:24.800310 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.800225 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13a1b6fa-ab80-4027-981f-5daf44a44b61-trusted-ca\") pod \"13a1b6fa-ab80-4027-981f-5daf44a44b61\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " Sep 19 10:36:24.800310 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.800269 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13a1b6fa-ab80-4027-981f-5daf44a44b61-ca-trust-extracted\") pod \"13a1b6fa-ab80-4027-981f-5daf44a44b61\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " Sep 19 10:36:24.800310 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.800286 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-certificates\") pod \"13a1b6fa-ab80-4027-981f-5daf44a44b61\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " Sep 19 10:36:24.800576 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.800314 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls\") pod \"13a1b6fa-ab80-4027-981f-5daf44a44b61\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " Sep 19 10:36:24.800576 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.800351 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13a1b6fa-ab80-4027-981f-5daf44a44b61-installation-pull-secrets\") pod \"13a1b6fa-ab80-4027-981f-5daf44a44b61\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " Sep 19 10:36:24.800576 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.800396 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13a1b6fa-ab80-4027-981f-5daf44a44b61-image-registry-private-configuration\") pod \"13a1b6fa-ab80-4027-981f-5daf44a44b61\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " Sep 19 10:36:24.800576 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.800479 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-bound-sa-token\") pod \"13a1b6fa-ab80-4027-981f-5daf44a44b61\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " Sep 19 10:36:24.800576 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.800516 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlp2l\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-kube-api-access-tlp2l\") pod \"13a1b6fa-ab80-4027-981f-5daf44a44b61\" (UID: \"13a1b6fa-ab80-4027-981f-5daf44a44b61\") " Sep 19 10:36:24.800833 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.800731 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "13a1b6fa-ab80-4027-981f-5daf44a44b61" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:36:24.801040 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.801005 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a1b6fa-ab80-4027-981f-5daf44a44b61-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "13a1b6fa-ab80-4027-981f-5daf44a44b61" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:36:24.803191 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.803137 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-kube-api-access-tlp2l" (OuterVolumeSpecName: "kube-api-access-tlp2l") pod "13a1b6fa-ab80-4027-981f-5daf44a44b61" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61"). InnerVolumeSpecName "kube-api-access-tlp2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:36:24.803483 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.803452 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "13a1b6fa-ab80-4027-981f-5daf44a44b61" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:36:24.803601 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.803536 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "13a1b6fa-ab80-4027-981f-5daf44a44b61" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:36:24.803601 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.803570 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a1b6fa-ab80-4027-981f-5daf44a44b61-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "13a1b6fa-ab80-4027-981f-5daf44a44b61" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 10:36:24.803721 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.803648 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a1b6fa-ab80-4027-981f-5daf44a44b61-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "13a1b6fa-ab80-4027-981f-5daf44a44b61" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 10:36:24.808355 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.808333 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a1b6fa-ab80-4027-981f-5daf44a44b61-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "13a1b6fa-ab80-4027-981f-5daf44a44b61" (UID: "13a1b6fa-ab80-4027-981f-5daf44a44b61"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:36:24.887687 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.887654 2579 generic.go:356] "Generic (PLEG): container finished" podID="13a1b6fa-ab80-4027-981f-5daf44a44b61" containerID="c86b4aa01dfeda78b8e71e6545f3c7f9ea8b7acaa64d0c3e244fa00284b5ca95" exitCode=0 Sep 19 10:36:24.887829 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.887697 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" Sep 19 10:36:24.887829 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.887699 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" event={"ID":"13a1b6fa-ab80-4027-981f-5daf44a44b61","Type":"ContainerDied","Data":"c86b4aa01dfeda78b8e71e6545f3c7f9ea8b7acaa64d0c3e244fa00284b5ca95"} Sep 19 10:36:24.887829 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.887747 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7774cfc878-x5gcd" event={"ID":"13a1b6fa-ab80-4027-981f-5daf44a44b61","Type":"ContainerDied","Data":"2e0f8c6b3d591b0c788ee7c8a0e765fe78141228310fcbb555eb2759850b8731"} Sep 19 10:36:24.887829 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.887775 2579 scope.go:117] "RemoveContainer" containerID="c86b4aa01dfeda78b8e71e6545f3c7f9ea8b7acaa64d0c3e244fa00284b5ca95" Sep 19 10:36:24.896491 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.896475 2579 scope.go:117] "RemoveContainer" containerID="c86b4aa01dfeda78b8e71e6545f3c7f9ea8b7acaa64d0c3e244fa00284b5ca95" Sep 19 10:36:24.896764 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:36:24.896745 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86b4aa01dfeda78b8e71e6545f3c7f9ea8b7acaa64d0c3e244fa00284b5ca95\": container with ID starting with c86b4aa01dfeda78b8e71e6545f3c7f9ea8b7acaa64d0c3e244fa00284b5ca95 not found: ID does not exist" containerID="c86b4aa01dfeda78b8e71e6545f3c7f9ea8b7acaa64d0c3e244fa00284b5ca95" Sep 19 10:36:24.896807 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.896779 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86b4aa01dfeda78b8e71e6545f3c7f9ea8b7acaa64d0c3e244fa00284b5ca95"} err="failed to get container status \"c86b4aa01dfeda78b8e71e6545f3c7f9ea8b7acaa64d0c3e244fa00284b5ca95\": rpc error: code = NotFound desc = could not find container \"c86b4aa01dfeda78b8e71e6545f3c7f9ea8b7acaa64d0c3e244fa00284b5ca95\": container with ID starting with c86b4aa01dfeda78b8e71e6545f3c7f9ea8b7acaa64d0c3e244fa00284b5ca95 not found: ID does not exist" Sep 19 10:36:24.901374 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.901358 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-bound-sa-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:36:24.901431 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.901376 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlp2l\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-kube-api-access-tlp2l\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:36:24.901431 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.901385 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13a1b6fa-ab80-4027-981f-5daf44a44b61-trusted-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:36:24.901431 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.901394 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13a1b6fa-ab80-4027-981f-5daf44a44b61-ca-trust-extracted\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:36:24.901431 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.901402 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-certificates\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:36:24.901431 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.901410 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a1b6fa-ab80-4027-981f-5daf44a44b61-registry-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:36:24.901431 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.901418 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13a1b6fa-ab80-4027-981f-5daf44a44b61-installation-pull-secrets\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:36:24.901431 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.901427 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13a1b6fa-ab80-4027-981f-5daf44a44b61-image-registry-private-configuration\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:36:24.909555 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.909508 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7774cfc878-x5gcd"] Sep 19 10:36:24.930109 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:24.930070 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7774cfc878-x5gcd"] Sep 19 10:36:26.195195 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:26.195162 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13a1b6fa-ab80-4027-981f-5daf44a44b61" path="/var/lib/kubelet/pods/13a1b6fa-ab80-4027-981f-5daf44a44b61/volumes" Sep 19 10:36:29.908663 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:29.908630 2579 generic.go:356] "Generic (PLEG): container finished" podID="a1e60166-09e7-42c8-aaea-c82942348bf7" containerID="df69cd9d9d30480e4bb4f7c7d7cdd6a2d45473a1fbf95c356b8c71f35b8e1198" exitCode=0 Sep 19 10:36:29.909050 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:29.908671 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" event={"ID":"a1e60166-09e7-42c8-aaea-c82942348bf7","Type":"ContainerDied","Data":"df69cd9d9d30480e4bb4f7c7d7cdd6a2d45473a1fbf95c356b8c71f35b8e1198"} Sep 19 10:36:29.909050 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:29.908923 2579 scope.go:117] "RemoveContainer" containerID="df69cd9d9d30480e4bb4f7c7d7cdd6a2d45473a1fbf95c356b8c71f35b8e1198" Sep 19 10:36:30.914777 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:30.914740 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-nn5f8" event={"ID":"a1e60166-09e7-42c8-aaea-c82942348bf7","Type":"ContainerStarted","Data":"a83758b1e2da72d7b8da56e911f64f6693d6aa5535e06209ae777981c821992b"} Sep 19 10:36:30.916205 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:30.916177 2579 generic.go:356] "Generic (PLEG): container finished" podID="efe9b96b-1362-4fc4-8701-36225c008ec9" containerID="56788311fc6e6fff0422e18c836a1a94333e64f0247cd52a415dbd98fd93d1f0" exitCode=0 Sep 19 10:36:30.916339 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:30.916219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" event={"ID":"efe9b96b-1362-4fc4-8701-36225c008ec9","Type":"ContainerDied","Data":"56788311fc6e6fff0422e18c836a1a94333e64f0247cd52a415dbd98fd93d1f0"} Sep 19 10:36:30.916555 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:30.916539 2579 scope.go:117] "RemoveContainer" containerID="56788311fc6e6fff0422e18c836a1a94333e64f0247cd52a415dbd98fd93d1f0" Sep 19 10:36:31.922424 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:31.922384 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-t5cft" event={"ID":"efe9b96b-1362-4fc4-8701-36225c008ec9","Type":"ContainerStarted","Data":"74044771b8b062bb2cc9579dc52b6563ad7dd39cbf3a4f700b4d4b223b00621f"} Sep 19 10:36:39.956454 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:39.956428 2579 generic.go:356] "Generic (PLEG): container finished" podID="bdae529f-3e44-4813-906d-fafa5aa85f27" containerID="68ce6f0b0f819e5720cbbace3139585e054a7a26a5cc62f9411b7ace86afb873" exitCode=0 Sep 19 10:36:39.956785 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:39.956504 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" event={"ID":"bdae529f-3e44-4813-906d-fafa5aa85f27","Type":"ContainerDied","Data":"68ce6f0b0f819e5720cbbace3139585e054a7a26a5cc62f9411b7ace86afb873"} Sep 19 10:36:39.956835 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:39.956826 2579 scope.go:117] "RemoveContainer" containerID="68ce6f0b0f819e5720cbbace3139585e054a7a26a5cc62f9411b7ace86afb873" Sep 19 10:36:40.962508 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:40.962477 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-2jc62" event={"ID":"bdae529f-3e44-4813-906d-fafa5aa85f27","Type":"ContainerStarted","Data":"7144c895ff4c1b27ec72d813b062275256099f8bb34b6defdfd8fae365cc2c71"} Sep 19 10:36:57.092702 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:57.092665 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:36:57.108204 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:57.108169 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:36:58.050616 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:36:58.050584 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:15.157413 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.157373 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 19 10:37:15.157936 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.157655 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="prometheus" containerID="cri-o://56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb" gracePeriod=600 Sep 19 10:37:15.157936 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.157703 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="thanos-sidecar" containerID="cri-o://f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128" gracePeriod=600 Sep 19 10:37:15.157936 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.157721 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="kube-rbac-proxy-web" containerID="cri-o://89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b" gracePeriod=600 Sep 19 10:37:15.157936 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.157746 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911" gracePeriod=600 Sep 19 10:37:15.157936 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.157759 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="config-reloader" containerID="cri-o://c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765" gracePeriod=600 Sep 19 10:37:15.157936 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.157703 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="kube-rbac-proxy" containerID="cri-o://89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87" gracePeriod=600 Sep 19 10:37:15.406513 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.406485 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:15.496738 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.496663 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-db\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.496738 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.496706 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-serving-certs-ca-bundle\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.496738 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.496735 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4clfz\" (UniqueName: \"kubernetes.io/projected/0914fa69-fbd9-4be4-8189-95708c8ad9c6-kube-api-access-4clfz\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.497020 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.496761 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-0\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.497020 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.496784 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-grpc-tls\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.497213 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.496813 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-trusted-ca-bundle\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.497409 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.497105 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:37:15.497506 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.497431 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:37:15.497606 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.497354 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-web-config\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.497689 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.497663 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:37:15.497807 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.497789 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-metrics-client-certs\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.497948 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.497930 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-1\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.498114 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.498099 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0914fa69-fbd9-4be4-8189-95708c8ad9c6-config-out\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.498517 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.497869 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:37:15.498620 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.498475 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:37:15.498731 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.498711 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-thanos-prometheus-http-client-file\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.498853 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.498839 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-config\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.498964 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.498949 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.499045 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.499034 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-metrics-client-ca\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.499164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.499148 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-2\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.499282 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.499268 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.499497 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.499480 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-kube-rbac-proxy\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.499877 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.499860 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-kubelet-serving-ca-bundle\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.500007 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.499992 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-tls\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.500138 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.500125 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0914fa69-fbd9-4be4-8189-95708c8ad9c6-tls-assets\") pod \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\" (UID: \"0914fa69-fbd9-4be4-8189-95708c8ad9c6\") " Sep 19 10:37:15.500495 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.499571 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:37:15.500495 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.499656 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:37:15.500635 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.500404 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 10:37:15.500635 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.500621 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:37:15.500743 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.500711 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0914fa69-fbd9-4be4-8189-95708c8ad9c6-kube-api-access-4clfz" (OuterVolumeSpecName: "kube-api-access-4clfz") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "kube-api-access-4clfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:37:15.500870 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.500844 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0914fa69-fbd9-4be4-8189-95708c8ad9c6-config-out" (OuterVolumeSpecName: "config-out") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:37:15.501084 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.501054 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.501200 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.501186 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-db\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.501294 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.501280 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.501380 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.501368 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.501462 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.501451 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.502883 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.502853 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 10:37:15.503286 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.503259 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-config" (OuterVolumeSpecName: "config") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 10:37:15.503934 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.503883 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 10:37:15.503934 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.503907 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 10:37:15.504283 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.504252 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 10:37:15.504816 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.504791 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 10:37:15.505221 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.505193 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 10:37:15.505829 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.505804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0914fa69-fbd9-4be4-8189-95708c8ad9c6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:37:15.515124 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.515092 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-web-config" (OuterVolumeSpecName: "web-config") pod "0914fa69-fbd9-4be4-8189-95708c8ad9c6" (UID: "0914fa69-fbd9-4be4-8189-95708c8ad9c6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 10:37:15.602203 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602171 2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-metrics-client-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602203 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602195 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0914fa69-fbd9-4be4-8189-95708c8ad9c6-config-out\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602203 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602206 2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602373 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602217 2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-config\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602373 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602226 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602373 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602237 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-metrics-client-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602373 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602245 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602373 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602254 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602373 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602263 2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-kube-rbac-proxy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602373 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602271 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0914fa69-fbd9-4be4-8189-95708c8ad9c6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602373 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602279 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602373 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602287 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0914fa69-fbd9-4be4-8189-95708c8ad9c6-tls-assets\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602373 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602294 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4clfz\" (UniqueName: \"kubernetes.io/projected/0914fa69-fbd9-4be4-8189-95708c8ad9c6-kube-api-access-4clfz\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602373 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602302 2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-secret-grpc-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:15.602373 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:15.602309 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0914fa69-fbd9-4be4-8189-95708c8ad9c6-web-config\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:37:16.107486 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107455 2579 generic.go:356] "Generic (PLEG): container finished" podID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerID="9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911" exitCode=0 Sep 19 10:37:16.107486 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107480 2579 generic.go:356] "Generic (PLEG): container finished" podID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerID="89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87" exitCode=0 Sep 19 10:37:16.107486 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107487 2579 generic.go:356] "Generic (PLEG): container finished" podID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerID="89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b" exitCode=0 Sep 19 10:37:16.107486 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107494 2579 generic.go:356] "Generic (PLEG): container finished" podID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerID="f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128" exitCode=0 Sep 19 10:37:16.107748 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107499 2579 generic.go:356] "Generic (PLEG): container finished" podID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerID="c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765" exitCode=0 Sep 19 10:37:16.107748 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107504 2579 generic.go:356] "Generic (PLEG): container finished" podID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerID="56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb" exitCode=0 Sep 19 10:37:16.107748 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107541 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.107748 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107540 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerDied","Data":"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911"} Sep 19 10:37:16.107748 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerDied","Data":"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87"} Sep 19 10:37:16.107748 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerDied","Data":"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b"} Sep 19 10:37:16.107968 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerDied","Data":"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128"} Sep 19 10:37:16.107968 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107770 2579 scope.go:117] "RemoveContainer" containerID="9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911" Sep 19 10:37:16.107968 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107787 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerDied","Data":"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765"} Sep 19 10:37:16.107968 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107813 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerDied","Data":"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb"} Sep 19 10:37:16.107968 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.107836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0914fa69-fbd9-4be4-8189-95708c8ad9c6","Type":"ContainerDied","Data":"e4a4c745d6307ed72b65bb00bb8a52beea880f8880497eb48668b8e0c8cfd2dc"} Sep 19 10:37:16.116419 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.116402 2579 scope.go:117] "RemoveContainer" containerID="89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87" Sep 19 10:37:16.123241 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.123221 2579 scope.go:117] "RemoveContainer" containerID="89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b" Sep 19 10:37:16.129415 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.129399 2579 scope.go:117] "RemoveContainer" containerID="f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128" Sep 19 10:37:16.134399 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.134377 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 19 10:37:16.137170 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.137145 2579 scope.go:117] "RemoveContainer" containerID="c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765" Sep 19 10:37:16.138000 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.137980 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 19 10:37:16.143968 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.143949 2579 scope.go:117] "RemoveContainer" containerID="56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb" Sep 19 10:37:16.150725 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.150704 2579 scope.go:117] "RemoveContainer" containerID="14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf" Sep 19 10:37:16.157327 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.157312 2579 scope.go:117] "RemoveContainer" containerID="9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911" Sep 19 10:37:16.157588 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:37:16.157570 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911\": container with ID starting with 9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911 not found: ID does not exist" containerID="9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911" Sep 19 10:37:16.157837 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.157597 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911"} err="failed to get container status \"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911\": rpc error: code = NotFound desc = could not find container \"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911\": container with ID starting with 9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911 not found: ID does not exist" Sep 19 10:37:16.157837 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.157616 2579 scope.go:117] "RemoveContainer" containerID="89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87" Sep 19 10:37:16.157909 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:37:16.157838 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87\": container with ID starting with 89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87 not found: ID does not exist" containerID="89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87" Sep 19 10:37:16.157909 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.157856 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87"} err="failed to get container status \"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87\": rpc error: code = NotFound desc = could not find container \"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87\": container with ID starting with 89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87 not found: ID does not exist" Sep 19 10:37:16.157909 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.157869 2579 scope.go:117] "RemoveContainer" containerID="89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b" Sep 19 10:37:16.158122 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:37:16.158105 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b\": container with ID starting with 89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b not found: ID does not exist" containerID="89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b" Sep 19 10:37:16.158169 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.158128 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b"} err="failed to get container status \"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b\": rpc error: code = NotFound desc = could not find container \"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b\": container with ID starting with 89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b not found: ID does not exist" Sep 19 10:37:16.158169 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.158142 2579 scope.go:117] "RemoveContainer" containerID="f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128" Sep 19 10:37:16.158383 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:37:16.158367 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128\": container with ID starting with f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128 not found: ID does not exist" containerID="f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128" Sep 19 10:37:16.158437 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.158387 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128"} err="failed to get container status \"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128\": rpc error: code = NotFound desc = could not find container \"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128\": container with ID starting with f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128 not found: ID does not exist" Sep 19 10:37:16.158437 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.158401 2579 scope.go:117] "RemoveContainer" containerID="c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765" Sep 19 10:37:16.158635 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:37:16.158619 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765\": container with ID starting with c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765 not found: ID does not exist" containerID="c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765" Sep 19 10:37:16.158675 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.158639 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765"} err="failed to get container status \"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765\": rpc error: code = NotFound desc = could not find container \"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765\": container with ID starting with c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765 not found: ID does not exist" Sep 19 10:37:16.158675 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.158652 2579 scope.go:117] "RemoveContainer" containerID="56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb" Sep 19 10:37:16.158880 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:37:16.158865 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb\": container with ID starting with 56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb not found: ID does not exist" containerID="56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb" Sep 19 10:37:16.158920 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.158882 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb"} err="failed to get container status \"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb\": rpc error: code = NotFound desc = could not find container \"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb\": container with ID starting with 56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb not found: ID does not exist" Sep 19 10:37:16.158920 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.158893 2579 scope.go:117] "RemoveContainer" containerID="14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf" Sep 19 10:37:16.159126 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:37:16.159109 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf\": container with ID starting with 14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf not found: ID does not exist" containerID="14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf" Sep 19 10:37:16.159177 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.159129 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf"} err="failed to get container status \"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf\": rpc error: code = NotFound desc = could not find container \"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf\": container with ID starting with 14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf not found: ID does not exist" Sep 19 10:37:16.159177 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.159141 2579 scope.go:117] "RemoveContainer" containerID="9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911" Sep 19 10:37:16.159389 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.159371 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911"} err="failed to get container status \"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911\": rpc error: code = NotFound desc = could not find container \"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911\": container with ID starting with 9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911 not found: ID does not exist" Sep 19 10:37:16.159436 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.159390 2579 scope.go:117] "RemoveContainer" containerID="89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87" Sep 19 10:37:16.159582 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.159565 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87"} err="failed to get container status \"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87\": rpc error: code = NotFound desc = could not find container \"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87\": container with ID starting with 89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87 not found: ID does not exist" Sep 19 10:37:16.159624 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.159583 2579 scope.go:117] "RemoveContainer" containerID="89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b" Sep 19 10:37:16.159766 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.159750 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b"} err="failed to get container status \"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b\": rpc error: code = NotFound desc = could not find container \"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b\": container with ID starting with 89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b not found: ID does not exist" Sep 19 10:37:16.159805 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.159766 2579 scope.go:117] "RemoveContainer" containerID="f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128" Sep 19 10:37:16.159958 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.159944 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128"} err="failed to get container status \"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128\": rpc error: code = NotFound desc = could not find container \"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128\": container with ID starting with f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128 not found: ID does not exist" Sep 19 10:37:16.159996 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.159959 2579 scope.go:117] "RemoveContainer" containerID="c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765" Sep 19 10:37:16.160159 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.160144 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765"} err="failed to get container status \"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765\": rpc error: code = NotFound desc = could not find container \"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765\": container with ID starting with c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765 not found: ID does not exist" Sep 19 10:37:16.160208 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.160159 2579 scope.go:117] "RemoveContainer" containerID="56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb" Sep 19 10:37:16.160382 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.160360 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb"} err="failed to get container status \"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb\": rpc error: code = NotFound desc = could not find container \"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb\": container with ID starting with 56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb not found: ID does not exist" Sep 19 10:37:16.160432 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.160383 2579 scope.go:117] "RemoveContainer" containerID="14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf" Sep 19 10:37:16.160566 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.160549 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf"} err="failed to get container status \"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf\": rpc error: code = NotFound desc = could not find container \"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf\": container with ID starting with 14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf not found: ID does not exist" Sep 19 10:37:16.160607 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.160565 2579 scope.go:117] "RemoveContainer" containerID="9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911" Sep 19 10:37:16.160792 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.160774 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911"} err="failed to get container status \"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911\": rpc error: code = NotFound desc = could not find container \"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911\": container with ID starting with 9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911 not found: ID does not exist" Sep 19 10:37:16.160833 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.160801 2579 scope.go:117] "RemoveContainer" containerID="89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87" Sep 19 10:37:16.161009 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.160988 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87"} err="failed to get container status \"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87\": rpc error: code = NotFound desc = could not find container \"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87\": container with ID starting with 89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87 not found: ID does not exist" Sep 19 10:37:16.161048 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.161010 2579 scope.go:117] "RemoveContainer" containerID="89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b" Sep 19 10:37:16.161250 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.161232 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b"} err="failed to get container status \"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b\": rpc error: code = NotFound desc = could not find container \"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b\": container with ID starting with 89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b not found: ID does not exist" Sep 19 10:37:16.161291 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.161250 2579 scope.go:117] "RemoveContainer" containerID="f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128" Sep 19 10:37:16.161458 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.161441 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128"} err="failed to get container status \"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128\": rpc error: code = NotFound desc = could not find container \"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128\": container with ID starting with f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128 not found: ID does not exist" Sep 19 10:37:16.161514 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.161458 2579 scope.go:117] "RemoveContainer" containerID="c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765" Sep 19 10:37:16.161655 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.161640 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765"} err="failed to get container status \"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765\": rpc error: code = NotFound desc = could not find container \"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765\": container with ID starting with c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765 not found: ID does not exist" Sep 19 10:37:16.161655 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.161654 2579 scope.go:117] "RemoveContainer" containerID="56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb" Sep 19 10:37:16.161845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.161831 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb"} err="failed to get container status \"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb\": rpc error: code = NotFound desc = could not find container \"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb\": container with ID starting with 56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb not found: ID does not exist" Sep 19 10:37:16.161885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.161845 2579 scope.go:117] "RemoveContainer" containerID="14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf" Sep 19 10:37:16.162028 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.162013 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf"} err="failed to get container status \"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf\": rpc error: code = NotFound desc = could not find container \"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf\": container with ID starting with 14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf not found: ID does not exist" Sep 19 10:37:16.162107 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.162028 2579 scope.go:117] "RemoveContainer" containerID="9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911" Sep 19 10:37:16.162273 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.162254 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911"} err="failed to get container status \"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911\": rpc error: code = NotFound desc = could not find container \"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911\": container with ID starting with 9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911 not found: ID does not exist" Sep 19 10:37:16.162314 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.162274 2579 scope.go:117] "RemoveContainer" containerID="89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87" Sep 19 10:37:16.162479 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.162463 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87"} err="failed to get container status \"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87\": rpc error: code = NotFound desc = could not find container \"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87\": container with ID starting with 89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87 not found: ID does not exist" Sep 19 10:37:16.162515 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.162480 2579 scope.go:117] "RemoveContainer" containerID="89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b" Sep 19 10:37:16.162728 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.162713 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b"} err="failed to get container status \"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b\": rpc error: code = NotFound desc = could not find container \"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b\": container with ID starting with 89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b not found: ID does not exist" Sep 19 10:37:16.162774 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.162727 2579 scope.go:117] "RemoveContainer" containerID="f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128" Sep 19 10:37:16.162924 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.162909 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128"} err="failed to get container status \"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128\": rpc error: code = NotFound desc = could not find container \"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128\": container with ID starting with f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128 not found: ID does not exist" Sep 19 10:37:16.162964 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.162924 2579 scope.go:117] "RemoveContainer" containerID="c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765" Sep 19 10:37:16.163168 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.163148 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765"} err="failed to get container status \"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765\": rpc error: code = NotFound desc = could not find container \"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765\": container with ID starting with c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765 not found: ID does not exist" Sep 19 10:37:16.163168 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.163167 2579 scope.go:117] "RemoveContainer" containerID="56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb" Sep 19 10:37:16.163351 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.163337 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb"} err="failed to get container status \"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb\": rpc error: code = NotFound desc = could not find container \"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb\": container with ID starting with 56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb not found: ID does not exist" Sep 19 10:37:16.163405 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.163351 2579 scope.go:117] "RemoveContainer" containerID="14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf" Sep 19 10:37:16.163597 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.163573 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf"} err="failed to get container status \"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf\": rpc error: code = NotFound desc = could not find container \"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf\": container with ID starting with 14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf not found: ID does not exist" Sep 19 10:37:16.163597 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.163596 2579 scope.go:117] "RemoveContainer" containerID="9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911" Sep 19 10:37:16.163872 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.163846 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911"} err="failed to get container status \"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911\": rpc error: code = NotFound desc = could not find container \"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911\": container with ID starting with 9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911 not found: ID does not exist" Sep 19 10:37:16.163975 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.163873 2579 scope.go:117] "RemoveContainer" containerID="89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87" Sep 19 10:37:16.164234 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.164197 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87"} err="failed to get container status \"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87\": rpc error: code = NotFound desc = could not find container \"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87\": container with ID starting with 89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87 not found: ID does not exist" Sep 19 10:37:16.164234 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.164222 2579 scope.go:117] "RemoveContainer" containerID="89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b" Sep 19 10:37:16.165430 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.165371 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b"} err="failed to get container status \"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b\": rpc error: code = NotFound desc = could not find container \"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b\": container with ID starting with 89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b not found: ID does not exist" Sep 19 10:37:16.165430 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.165397 2579 scope.go:117] "RemoveContainer" containerID="f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128" Sep 19 10:37:16.165682 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.165650 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128"} err="failed to get container status \"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128\": rpc error: code = NotFound desc = could not find container \"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128\": container with ID starting with f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128 not found: ID does not exist" Sep 19 10:37:16.165682 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.165673 2579 scope.go:117] "RemoveContainer" containerID="c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765" Sep 19 10:37:16.165985 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.165961 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765"} err="failed to get container status \"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765\": rpc error: code = NotFound desc = could not find container \"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765\": container with ID starting with c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765 not found: ID does not exist" Sep 19 10:37:16.166086 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.165985 2579 scope.go:117] "RemoveContainer" containerID="56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb" Sep 19 10:37:16.166155 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166122 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 19 10:37:16.166295 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166271 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb"} err="failed to get container status \"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb\": rpc error: code = NotFound desc = could not find container \"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb\": container with ID starting with 56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb not found: ID does not exist" Sep 19 10:37:16.166295 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166294 2579 scope.go:117] "RemoveContainer" containerID="14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf" Sep 19 10:37:16.166505 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166487 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="config-reloader" Sep 19 10:37:16.166577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166512 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="config-reloader" Sep 19 10:37:16.166577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166525 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="thanos-sidecar" Sep 19 10:37:16.166577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166532 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="thanos-sidecar" Sep 19 10:37:16.166577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166548 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="kube-rbac-proxy-web" Sep 19 10:37:16.166577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166556 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="kube-rbac-proxy-web" Sep 19 10:37:16.166577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166573 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="init-config-reloader" Sep 19 10:37:16.166577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166578 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="init-config-reloader" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166589 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="prometheus" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166594 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="prometheus" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166608 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="kube-rbac-proxy" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166614 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="kube-rbac-proxy" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166624 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a1b6fa-ab80-4027-981f-5daf44a44b61" containerName="registry" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166631 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a1b6fa-ab80-4027-981f-5daf44a44b61" containerName="registry" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166640 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="kube-rbac-proxy-thanos" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166648 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="kube-rbac-proxy-thanos" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166709 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="kube-rbac-proxy-thanos" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166721 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="prometheus" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166732 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="thanos-sidecar" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166742 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="kube-rbac-proxy-web" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166750 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="kube-rbac-proxy" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166759 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a1b6fa-ab80-4027-981f-5daf44a44b61" containerName="registry" Sep 19 10:37:16.166885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.166767 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" containerName="config-reloader" Sep 19 10:37:16.167819 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.167733 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf"} err="failed to get container status \"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf\": rpc error: code = NotFound desc = could not find container \"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf\": container with ID starting with 14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf not found: ID does not exist" Sep 19 10:37:16.167819 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.167757 2579 scope.go:117] "RemoveContainer" containerID="9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911" Sep 19 10:37:16.168245 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.168162 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911"} err="failed to get container status \"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911\": rpc error: code = NotFound desc = could not find container \"9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911\": container with ID starting with 9ab52e8534a55bd7abd3dc65fe298a3bcc191769333050b590cc6c96e0f38911 not found: ID does not exist" Sep 19 10:37:16.168245 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.168183 2579 scope.go:117] "RemoveContainer" containerID="89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87" Sep 19 10:37:16.168674 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.168554 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87"} err="failed to get container status \"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87\": rpc error: code = NotFound desc = could not find container \"89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87\": container with ID starting with 89e18b0208360ea87fed666f4dd7065ba5a35e2fa52ab2cb1b9265acb94d5b87 not found: ID does not exist" Sep 19 10:37:16.168674 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.168609 2579 scope.go:117] "RemoveContainer" containerID="89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b" Sep 19 10:37:16.169052 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.168974 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b"} err="failed to get container status \"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b\": rpc error: code = NotFound desc = could not find container \"89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b\": container with ID starting with 89a671ecf708f75f9ef778271c01d3fbb4c24501b759516a4d4d5bf280457b2b not found: ID does not exist" Sep 19 10:37:16.169052 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.168996 2579 scope.go:117] "RemoveContainer" containerID="f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128" Sep 19 10:37:16.169456 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.169376 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128"} err="failed to get container status \"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128\": rpc error: code = NotFound desc = could not find container \"f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128\": container with ID starting with f8944c2f0d8df8e4342e45f3e9759e24e09ebbae4e99ab1b85a45b4827da5128 not found: ID does not exist" Sep 19 10:37:16.169456 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.169396 2579 scope.go:117] "RemoveContainer" containerID="c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765" Sep 19 10:37:16.169833 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.169754 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765"} err="failed to get container status \"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765\": rpc error: code = NotFound desc = could not find container \"c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765\": container with ID starting with c42bb51209f7c27275e2ef59977b214287d5e6b9f73c18bdd83985541232c765 not found: ID does not exist" Sep 19 10:37:16.169833 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.169776 2579 scope.go:117] "RemoveContainer" containerID="56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb" Sep 19 10:37:16.170225 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.170145 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb"} err="failed to get container status \"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb\": rpc error: code = NotFound desc = could not find container \"56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb\": container with ID starting with 56731e169f492e184020c0c5155dd8f2a5a37634e346e619f87839dbaef472cb not found: ID does not exist" Sep 19 10:37:16.170225 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.170168 2579 scope.go:117] "RemoveContainer" containerID="14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf" Sep 19 10:37:16.170572 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.170543 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf"} err="failed to get container status \"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf\": rpc error: code = NotFound desc = could not find container \"14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf\": container with ID starting with 14744b948c6c95f5dc42d466ad0e9ad676344897faf277d67764a9fbb2d9f5cf not found: ID does not exist" Sep 19 10:37:16.173164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.173130 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.176053 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.175865 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 19 10:37:16.176053 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.175865 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 19 10:37:16.176649 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.176465 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 19 10:37:16.176649 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.176516 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 19 10:37:16.176649 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.176539 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 19 10:37:16.176649 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.176522 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 19 10:37:16.176649 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.176590 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 19 10:37:16.176891 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.176676 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 19 10:37:16.176941 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.176907 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 19 10:37:16.176994 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.176942 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hvv6k\"" Sep 19 10:37:16.177184 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.177163 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 19 10:37:16.177296 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.177229 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 19 10:37:16.177485 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.177450 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-42dugplt7ovcs\"" Sep 19 10:37:16.177572 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.177517 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 19 10:37:16.178064 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.178042 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 19 10:37:16.179795 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.179779 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 19 10:37:16.181348 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.181328 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 19 10:37:16.187472 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.187449 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 19 10:37:16.196046 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.196024 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0914fa69-fbd9-4be4-8189-95708c8ad9c6" path="/var/lib/kubelet/pods/0914fa69-fbd9-4be4-8189-95708c8ad9c6/volumes" Sep 19 10:37:16.306896 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.306868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.306896 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.306900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-config\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307123 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.306919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307123 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.306937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0d32aa7-c4a7-423f-836b-b348353e8dea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307123 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.306979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307123 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307022 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307123 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307089 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307123 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307367 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfnc\" (UniqueName: \"kubernetes.io/projected/e0d32aa7-c4a7-423f-836b-b348353e8dea-kube-api-access-6xfnc\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307367 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307367 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0d32aa7-c4a7-423f-836b-b348353e8dea-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307367 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307228 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307367 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307244 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307367 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307367 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307367 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307367 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307367 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307663 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.307663 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.307400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.407665 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.407639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.407792 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.407675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.407792 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.407697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.407792 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.407720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.407947 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.407895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.407947 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.407936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408114 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408183 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-config\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408183 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408281 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0d32aa7-c4a7-423f-836b-b348353e8dea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408281 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408281 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408281 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408481 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408481 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfnc\" (UniqueName: \"kubernetes.io/projected/e0d32aa7-c4a7-423f-836b-b348353e8dea-kube-api-access-6xfnc\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408481 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408481 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0d32aa7-c4a7-423f-836b-b348353e8dea-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408481 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408481 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408765 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408765 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408539 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.408765 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.408558 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.409378 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.409035 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.410794 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.410769 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.412164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.410882 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.412164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.411307 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0d32aa7-c4a7-423f-836b-b348353e8dea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.412164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.410882 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.412164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.411386 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.412164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.411506 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.412164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.411831 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.412164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.411898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.412164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.412100 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.412164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.412103 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.412523 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.412185 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0d32aa7-c4a7-423f-836b-b348353e8dea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.413166 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.413146 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-config\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.413288 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.413261 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d32aa7-c4a7-423f-836b-b348353e8dea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.413719 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.413699 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0d32aa7-c4a7-423f-836b-b348353e8dea-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.413991 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.413969 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.414189 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.414173 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0d32aa7-c4a7-423f-836b-b348353e8dea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.418743 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.418722 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfnc\" (UniqueName: \"kubernetes.io/projected/e0d32aa7-c4a7-423f-836b-b348353e8dea-kube-api-access-6xfnc\") pod \"prometheus-k8s-0\" (UID: \"e0d32aa7-c4a7-423f-836b-b348353e8dea\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.495486 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.495456 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:37:16.791759 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:16.791674 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 19 10:37:16.794475 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:37:16.794445 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d32aa7_c4a7_423f_836b_b348353e8dea.slice/crio-037be6a16e5b09c12aebbe22129a37b6c070228cb80d045cab01fa03d599c612 WatchSource:0}: Error finding container 037be6a16e5b09c12aebbe22129a37b6c070228cb80d045cab01fa03d599c612: Status 404 returned error can't find the container with id 037be6a16e5b09c12aebbe22129a37b6c070228cb80d045cab01fa03d599c612 Sep 19 10:37:17.112602 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:17.112531 2579 generic.go:356] "Generic (PLEG): container finished" podID="e0d32aa7-c4a7-423f-836b-b348353e8dea" containerID="34e9cfe3badb8a8a642af0356852ddb3eeca00f8da7515085c3debc365e4f581" exitCode=0 Sep 19 10:37:17.112742 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:17.112621 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d32aa7-c4a7-423f-836b-b348353e8dea","Type":"ContainerDied","Data":"34e9cfe3badb8a8a642af0356852ddb3eeca00f8da7515085c3debc365e4f581"} Sep 19 10:37:17.112742 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:17.112735 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d32aa7-c4a7-423f-836b-b348353e8dea","Type":"ContainerStarted","Data":"037be6a16e5b09c12aebbe22129a37b6c070228cb80d045cab01fa03d599c612"} Sep 19 10:37:18.120952 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:18.120913 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d32aa7-c4a7-423f-836b-b348353e8dea","Type":"ContainerStarted","Data":"1cdd6a894dd23622d0c7bb3c8aa0a6ebfd2395f6edea7eec77f53ac56a0a9e5f"} Sep 19 10:37:18.121421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:18.120997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d32aa7-c4a7-423f-836b-b348353e8dea","Type":"ContainerStarted","Data":"51425b0fb3d42852e59bf3f2dcb09b7de577b7d50117136905e5c28981c3797c"} Sep 19 10:37:18.121421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:18.121039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d32aa7-c4a7-423f-836b-b348353e8dea","Type":"ContainerStarted","Data":"9bdb23f8bbd9d21081927044b2f97fe089be1ae16716aba2c9bb03122e9eb3d1"} Sep 19 10:37:18.121421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:18.121097 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d32aa7-c4a7-423f-836b-b348353e8dea","Type":"ContainerStarted","Data":"ebdd2c6046a9905b566c574079c41cd7a417ac64345a94be0f0e9d961685f889"} Sep 19 10:37:18.121421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:18.121140 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d32aa7-c4a7-423f-836b-b348353e8dea","Type":"ContainerStarted","Data":"3f14cee2d0f900c7f7d91d8bdd3d1abace45ce01149ffad6edc70317fde83c04"} Sep 19 10:37:18.121421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:18.121177 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d32aa7-c4a7-423f-836b-b348353e8dea","Type":"ContainerStarted","Data":"20f979c0b7e601328cde4833a01f0fd700875303410235127b523701714bc53d"} Sep 19 10:37:18.152169 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:18.152117 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.134008219 podStartE2EDuration="2.152102506s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.094287ms" isStatefulPod=false podCreationTimestamp="2026-09-19 10:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:37:18.149583427 +0000 UTC m=+170.062088140" watchObservedRunningTime="2026-09-19 10:37:18.152102506 +0000 UTC m=+170.064607222" Sep 19 10:37:21.495965 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:37:21.495921 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:38:16.496135 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:38:16.496101 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:38:16.511020 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:38:16.510994 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:38:17.372017 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:38:17.371990 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 19 10:39:16.357689 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.357651 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf"] Sep 19 10:39:16.361051 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.361030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.363748 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.363719 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 19 10:39:16.363873 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.363773 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 19 10:39:16.364106 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.364084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bmfwr\"" Sep 19 10:39:16.364217 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.364110 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 19 10:39:16.364217 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.364131 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 19 10:39:16.374804 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.374785 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf"] Sep 19 10:39:16.431229 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.431207 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9aca349-9040-4593-989f-0fdc14b0fe0a-webhook-cert\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.431392 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.431248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9aca349-9040-4593-989f-0fdc14b0fe0a-manifests\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.431392 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.431279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9aca349-9040-4593-989f-0fdc14b0fe0a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.431505 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.431388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zjnc\" (UniqueName: \"kubernetes.io/projected/f9aca349-9040-4593-989f-0fdc14b0fe0a-kube-api-access-6zjnc\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.431505 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.431477 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9aca349-9040-4593-989f-0fdc14b0fe0a-tmp\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.532223 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.532194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zjnc\" (UniqueName: \"kubernetes.io/projected/f9aca349-9040-4593-989f-0fdc14b0fe0a-kube-api-access-6zjnc\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.532391 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.532265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9aca349-9040-4593-989f-0fdc14b0fe0a-tmp\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.532391 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.532294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9aca349-9040-4593-989f-0fdc14b0fe0a-webhook-cert\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.532391 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.532323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9aca349-9040-4593-989f-0fdc14b0fe0a-manifests\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.532391 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.532352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9aca349-9040-4593-989f-0fdc14b0fe0a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.532755 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.532732 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9aca349-9040-4593-989f-0fdc14b0fe0a-manifests\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.534315 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.534283 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9aca349-9040-4593-989f-0fdc14b0fe0a-tmp\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.534526 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.534508 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9aca349-9040-4593-989f-0fdc14b0fe0a-webhook-cert\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.534590 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.534532 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9aca349-9040-4593-989f-0fdc14b0fe0a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.543206 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.543184 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zjnc\" (UniqueName: \"kubernetes.io/projected/f9aca349-9040-4593-989f-0fdc14b0fe0a-kube-api-access-6zjnc\") pod \"opendatahub-operator-controller-manager-9b87889df-xw7hf\" (UID: \"f9aca349-9040-4593-989f-0fdc14b0fe0a\") " pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.671806 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.671781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:16.959932 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:16.959907 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf"] Sep 19 10:39:16.961797 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:39:16.961768 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9aca349_9040_4593_989f_0fdc14b0fe0a.slice/crio-f5a5a6770b9f5a07bf813ae74f2e97b75285d984707ef58d00856f0562c18939 WatchSource:0}: Error finding container f5a5a6770b9f5a07bf813ae74f2e97b75285d984707ef58d00856f0562c18939: Status 404 returned error can't find the container with id f5a5a6770b9f5a07bf813ae74f2e97b75285d984707ef58d00856f0562c18939 Sep 19 10:39:17.587537 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:17.587478 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" event={"ID":"f9aca349-9040-4593-989f-0fdc14b0fe0a","Type":"ContainerStarted","Data":"f5a5a6770b9f5a07bf813ae74f2e97b75285d984707ef58d00856f0562c18939"} Sep 19 10:39:20.605940 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:20.605903 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" event={"ID":"f9aca349-9040-4593-989f-0fdc14b0fe0a","Type":"ContainerStarted","Data":"207eb8faca211c05d057351a01d10a8a65e70627645cc83cd7b7a68320aea34e"} Sep 19 10:39:21.611695 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:21.611661 2579 generic.go:356] "Generic (PLEG): container finished" podID="f9aca349-9040-4593-989f-0fdc14b0fe0a" containerID="207eb8faca211c05d057351a01d10a8a65e70627645cc83cd7b7a68320aea34e" exitCode=0 Sep 19 10:39:21.612191 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:21.611715 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" event={"ID":"f9aca349-9040-4593-989f-0fdc14b0fe0a","Type":"ContainerDied","Data":"207eb8faca211c05d057351a01d10a8a65e70627645cc83cd7b7a68320aea34e"} Sep 19 10:39:22.617749 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:22.617711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" event={"ID":"f9aca349-9040-4593-989f-0fdc14b0fe0a","Type":"ContainerStarted","Data":"62005830bab3f8d43b778aa4b8c2d153b061b8ace31806196a79a026ed7bc78c"} Sep 19 10:39:22.618160 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:22.617860 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:22.642176 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:22.642135 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" podStartSLOduration=2.386892532 podStartE2EDuration="6.642122091s" totalImagesPullingTime="3.166197216s" totalInitContainerRuntime="1.089032343s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:39:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:39:22.641681159 +0000 UTC m=+294.554185877" watchObservedRunningTime="2026-09-19 10:39:22.642122091 +0000 UTC m=+294.554626806" Sep 19 10:39:30.136300 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:30.136266 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 10:39:30.142705 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:30.142683 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 10:39:30.147728 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:30.147707 2579 kubelet.go:1648] "Image garbage collection succeeded" Sep 19 10:39:33.624775 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:33.624746 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9b87889df-xw7hf" Sep 19 10:39:36.179155 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.179115 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5"] Sep 19 10:39:36.181137 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.181119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.183574 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.183550 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 19 10:39:36.183767 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.183753 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 19 10:39:36.184759 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.184742 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 19 10:39:36.184851 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.184838 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 19 10:39:36.184957 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.184935 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mm8vq\"" Sep 19 10:39:36.185096 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.184989 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 19 10:39:36.195890 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.195869 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5"] Sep 19 10:39:36.300475 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.300443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbcd0ddf-e8d9-4a89-b535-6ce173163f54-cert\") pod \"lws-controller-manager-6c7d4469d9-wpkp5\" (UID: \"fbcd0ddf-e8d9-4a89-b535-6ce173163f54\") " pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.300640 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.300482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fbcd0ddf-e8d9-4a89-b535-6ce173163f54-manager-config\") pod \"lws-controller-manager-6c7d4469d9-wpkp5\" (UID: \"fbcd0ddf-e8d9-4a89-b535-6ce173163f54\") " pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.300640 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.300562 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbcd0ddf-e8d9-4a89-b535-6ce173163f54-metrics-cert\") pod \"lws-controller-manager-6c7d4469d9-wpkp5\" (UID: \"fbcd0ddf-e8d9-4a89-b535-6ce173163f54\") " pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.300640 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.300593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmm85\" (UniqueName: \"kubernetes.io/projected/fbcd0ddf-e8d9-4a89-b535-6ce173163f54-kube-api-access-lmm85\") pod \"lws-controller-manager-6c7d4469d9-wpkp5\" (UID: \"fbcd0ddf-e8d9-4a89-b535-6ce173163f54\") " pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.401095 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.401047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbcd0ddf-e8d9-4a89-b535-6ce173163f54-metrics-cert\") pod \"lws-controller-manager-6c7d4469d9-wpkp5\" (UID: \"fbcd0ddf-e8d9-4a89-b535-6ce173163f54\") " pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.401232 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.401101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmm85\" (UniqueName: \"kubernetes.io/projected/fbcd0ddf-e8d9-4a89-b535-6ce173163f54-kube-api-access-lmm85\") pod \"lws-controller-manager-6c7d4469d9-wpkp5\" (UID: \"fbcd0ddf-e8d9-4a89-b535-6ce173163f54\") " pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.401232 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.401166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbcd0ddf-e8d9-4a89-b535-6ce173163f54-cert\") pod \"lws-controller-manager-6c7d4469d9-wpkp5\" (UID: \"fbcd0ddf-e8d9-4a89-b535-6ce173163f54\") " pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.401232 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.401191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fbcd0ddf-e8d9-4a89-b535-6ce173163f54-manager-config\") pod \"lws-controller-manager-6c7d4469d9-wpkp5\" (UID: \"fbcd0ddf-e8d9-4a89-b535-6ce173163f54\") " pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.401803 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.401781 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fbcd0ddf-e8d9-4a89-b535-6ce173163f54-manager-config\") pod \"lws-controller-manager-6c7d4469d9-wpkp5\" (UID: \"fbcd0ddf-e8d9-4a89-b535-6ce173163f54\") " pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.403558 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.403537 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbcd0ddf-e8d9-4a89-b535-6ce173163f54-cert\") pod \"lws-controller-manager-6c7d4469d9-wpkp5\" (UID: \"fbcd0ddf-e8d9-4a89-b535-6ce173163f54\") " pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.403635 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.403594 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbcd0ddf-e8d9-4a89-b535-6ce173163f54-metrics-cert\") pod \"lws-controller-manager-6c7d4469d9-wpkp5\" (UID: \"fbcd0ddf-e8d9-4a89-b535-6ce173163f54\") " pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.423404 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.423383 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmm85\" (UniqueName: \"kubernetes.io/projected/fbcd0ddf-e8d9-4a89-b535-6ce173163f54-kube-api-access-lmm85\") pod \"lws-controller-manager-6c7d4469d9-wpkp5\" (UID: \"fbcd0ddf-e8d9-4a89-b535-6ce173163f54\") " pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.490494 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.490439 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:36.765096 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.764968 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5"] Sep 19 10:39:36.768963 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:39:36.768936 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcd0ddf_e8d9_4a89_b535_6ce173163f54.slice/crio-4d7c7cb20c3e66c04909b2a7fa89e5efb9e254bb5211d8f3f3ed3869f5fb8cce WatchSource:0}: Error finding container 4d7c7cb20c3e66c04909b2a7fa89e5efb9e254bb5211d8f3f3ed3869f5fb8cce: Status 404 returned error can't find the container with id 4d7c7cb20c3e66c04909b2a7fa89e5efb9e254bb5211d8f3f3ed3869f5fb8cce Sep 19 10:39:36.770937 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:36.770920 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 19 10:39:37.681615 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:37.681574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" event={"ID":"fbcd0ddf-e8d9-4a89-b535-6ce173163f54","Type":"ContainerStarted","Data":"4d7c7cb20c3e66c04909b2a7fa89e5efb9e254bb5211d8f3f3ed3869f5fb8cce"} Sep 19 10:39:39.691624 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:39.691592 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" event={"ID":"fbcd0ddf-e8d9-4a89-b535-6ce173163f54","Type":"ContainerStarted","Data":"5386be025aa92101008950324eafbe468a62f633122b5692679d6f037bfe6e8c"} Sep 19 10:39:39.692143 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:39.691650 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:39.725473 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:39.725427 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" podStartSLOduration=1.528217416 podStartE2EDuration="3.725411444s" totalImagesPullingTime="2.197194028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:39:39.720626336 +0000 UTC m=+311.633131052" watchObservedRunningTime="2026-09-19 10:39:39.725411444 +0000 UTC m=+311.637916160" Sep 19 10:39:42.310348 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.310312 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6"] Sep 19 10:39:42.313927 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.313907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.316783 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.316762 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 19 10:39:42.316955 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.316933 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8xd97\"" Sep 19 10:39:42.331407 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.331385 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6"] Sep 19 10:39:42.378217 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.378187 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd"] Sep 19 10:39:42.381526 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.381506 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.394889 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.394864 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd"] Sep 19 10:39:42.454360 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.454479 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.454479 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.454479 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.454479 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.454613 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.454613 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454549 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.454613 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.454613 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hv2d\" (UniqueName: \"kubernetes.io/projected/da0da6f6-fe34-4d35-90b8-f8bc487946c0-kube-api-access-9hv2d\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.454613 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2965\" (UniqueName: \"kubernetes.io/projected/9026a8a4-9b07-4752-ab91-5c331a6fbeba-kube-api-access-w2965\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.454769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.454769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.454769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.454769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.454769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.454912 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.454912 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.454912 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.454823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.555795 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.555765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.555795 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.555799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.555989 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.555823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.555989 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.555855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.555989 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.555886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.555989 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.555918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.555989 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.555945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.556345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.556345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.556345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.556345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hv2d\" (UniqueName: \"kubernetes.io/projected/da0da6f6-fe34-4d35-90b8-f8bc487946c0-kube-api-access-9hv2d\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.556345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2965\" (UniqueName: \"kubernetes.io/projected/9026a8a4-9b07-4752-ab91-5c331a6fbeba-kube-api-access-w2965\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.556345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.556345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.556345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556255 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.556345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.556345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.556345 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.556883 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.556883 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556399 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.556883 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.556883 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556480 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.556883 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556533 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.556883 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556669 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.556883 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556280 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.556883 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.556697 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.557286 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.557138 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.557533 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.557507 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.558870 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.558847 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.558973 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.558903 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.558973 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.558923 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.558973 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.558945 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.564934 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.564876 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2965\" (UniqueName: \"kubernetes.io/projected/9026a8a4-9b07-4752-ab91-5c331a6fbeba-kube-api-access-w2965\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.565047 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.565026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.565361 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.565341 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da0da6f6-fe34-4d35-90b8-f8bc487946c0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.565912 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.565891 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hv2d\" (UniqueName: \"kubernetes.io/projected/da0da6f6-fe34-4d35-90b8-f8bc487946c0-kube-api-access-9hv2d\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd\" (UID: \"da0da6f6-fe34-4d35-90b8-f8bc487946c0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.624886 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.624861 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:42.691375 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.691148 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:42.929520 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.929495 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6"] Sep 19 10:39:42.931941 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:39:42.931917 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9026a8a4_9b07_4752_ab91_5c331a6fbeba.slice/crio-35bb394762bdf280417e87879bf4203a5b9e2a6b55f6e0d4ae3a871f261d8f5f WatchSource:0}: Error finding container 35bb394762bdf280417e87879bf4203a5b9e2a6b55f6e0d4ae3a871f261d8f5f: Status 404 returned error can't find the container with id 35bb394762bdf280417e87879bf4203a5b9e2a6b55f6e0d4ae3a871f261d8f5f Sep 19 10:39:42.971848 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:42.971745 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd"] Sep 19 10:39:42.973886 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:39:42.973859 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0da6f6_fe34_4d35_90b8_f8bc487946c0.slice/crio-3654c6e4f18bacfa39b5d863a07a9d741fd5d51641820848d6dca56d36a662f1 WatchSource:0}: Error finding container 3654c6e4f18bacfa39b5d863a07a9d741fd5d51641820848d6dca56d36a662f1: Status 404 returned error can't find the container with id 3654c6e4f18bacfa39b5d863a07a9d741fd5d51641820848d6dca56d36a662f1 Sep 19 10:39:43.713434 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:43.713400 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" event={"ID":"9026a8a4-9b07-4752-ab91-5c331a6fbeba","Type":"ContainerStarted","Data":"35bb394762bdf280417e87879bf4203a5b9e2a6b55f6e0d4ae3a871f261d8f5f"} Sep 19 10:39:43.716018 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:43.715934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" event={"ID":"da0da6f6-fe34-4d35-90b8-f8bc487946c0","Type":"ContainerStarted","Data":"3654c6e4f18bacfa39b5d863a07a9d741fd5d51641820848d6dca56d36a662f1"} Sep 19 10:39:45.638768 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:45.638719 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:39:45.639044 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:45.638803 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:39:45.639044 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:45.638828 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:39:45.644848 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:45.644815 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:39:45.644917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:45.644889 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:39:45.644956 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:45.644927 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:39:45.727001 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:45.726960 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" event={"ID":"da0da6f6-fe34-4d35-90b8-f8bc487946c0","Type":"ContainerStarted","Data":"2bcf8d364b03b918f4749a402da7bda5b77abe980ff51420137422b36c8f2ac8"} Sep 19 10:39:45.728927 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:45.728904 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" event={"ID":"9026a8a4-9b07-4752-ab91-5c331a6fbeba","Type":"ContainerStarted","Data":"afa4e75f542aa1d21cc2922379137d27d76cf7150d548c10df60990313661083"} Sep 19 10:39:45.750677 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:45.750611 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" podStartSLOduration=1.087972579 podStartE2EDuration="3.750590497s" totalImagesPullingTime="2.662617918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:39:45.74898974 +0000 UTC m=+317.661494458" watchObservedRunningTime="2026-09-19 10:39:45.750590497 +0000 UTC m=+317.663095214" Sep 19 10:39:45.772989 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:45.772918 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" podStartSLOduration=1.06225169 podStartE2EDuration="3.772896633s" totalImagesPullingTime="2.710644943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:39:45.771174707 +0000 UTC m=+317.683679424" watchObservedRunningTime="2026-09-19 10:39:45.772896633 +0000 UTC m=+317.685401351" Sep 19 10:39:46.237471 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.237438 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-m7hz5"] Sep 19 10:39:46.240790 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.240774 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" Sep 19 10:39:46.243528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.243507 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bzgbb\"" Sep 19 10:39:46.243937 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.243922 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 19 10:39:46.251422 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.251400 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-m7hz5"] Sep 19 10:39:46.392740 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.392713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwj5c\" (UniqueName: \"kubernetes.io/projected/9dd3d0c6-e1f0-4381-acfa-535576a57096-kube-api-access-qwj5c\") pod \"odh-model-controller-7998c67dc4-m7hz5\" (UID: \"9dd3d0c6-e1f0-4381-acfa-535576a57096\") " pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" Sep 19 10:39:46.392896 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.392748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dd3d0c6-e1f0-4381-acfa-535576a57096-cert\") pod \"odh-model-controller-7998c67dc4-m7hz5\" (UID: \"9dd3d0c6-e1f0-4381-acfa-535576a57096\") " pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" Sep 19 10:39:46.493488 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.493408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwj5c\" (UniqueName: \"kubernetes.io/projected/9dd3d0c6-e1f0-4381-acfa-535576a57096-kube-api-access-qwj5c\") pod \"odh-model-controller-7998c67dc4-m7hz5\" (UID: \"9dd3d0c6-e1f0-4381-acfa-535576a57096\") " pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" Sep 19 10:39:46.493488 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.493443 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dd3d0c6-e1f0-4381-acfa-535576a57096-cert\") pod \"odh-model-controller-7998c67dc4-m7hz5\" (UID: \"9dd3d0c6-e1f0-4381-acfa-535576a57096\") " pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" Sep 19 10:39:46.493659 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:39:46.493571 2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 19 10:39:46.493659 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:39:46.493622 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd3d0c6-e1f0-4381-acfa-535576a57096-cert podName:9dd3d0c6-e1f0-4381-acfa-535576a57096 nodeName:}" failed. No retries permitted until 2026-09-19 10:39:46.993606681 +0000 UTC m=+318.906111375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9dd3d0c6-e1f0-4381-acfa-535576a57096-cert") pod "odh-model-controller-7998c67dc4-m7hz5" (UID: "9dd3d0c6-e1f0-4381-acfa-535576a57096") : secret "odh-model-controller-webhook-cert" not found Sep 19 10:39:46.502118 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.502093 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwj5c\" (UniqueName: \"kubernetes.io/projected/9dd3d0c6-e1f0-4381-acfa-535576a57096-kube-api-access-qwj5c\") pod \"odh-model-controller-7998c67dc4-m7hz5\" (UID: \"9dd3d0c6-e1f0-4381-acfa-535576a57096\") " pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" Sep 19 10:39:46.625626 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.625595 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:46.627162 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.627124 2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Sep 19 10:39:46.627294 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.627201 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" podUID="9026a8a4-9b07-4752-ab91-5c331a6fbeba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Sep 19 10:39:46.692531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.692500 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:46.696814 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.696792 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:46.733873 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.733821 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:46.734754 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.734735 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd" Sep 19 10:39:46.804935 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.804846 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6"] Sep 19 10:39:46.997703 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:46.997667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dd3d0c6-e1f0-4381-acfa-535576a57096-cert\") pod \"odh-model-controller-7998c67dc4-m7hz5\" (UID: \"9dd3d0c6-e1f0-4381-acfa-535576a57096\") " pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" Sep 19 10:39:47.000115 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:47.000087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dd3d0c6-e1f0-4381-acfa-535576a57096-cert\") pod \"odh-model-controller-7998c67dc4-m7hz5\" (UID: \"9dd3d0c6-e1f0-4381-acfa-535576a57096\") " pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" Sep 19 10:39:47.151208 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:47.151178 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" Sep 19 10:39:47.426357 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:47.426196 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-m7hz5"] Sep 19 10:39:47.429136 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:39:47.429102 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd3d0c6_e1f0_4381_acfa_535576a57096.slice/crio-85a234e207dac2b93d9497d4f1ea8a98dd474bd28dcc9bdc15cc1f253166688f WatchSource:0}: Error finding container 85a234e207dac2b93d9497d4f1ea8a98dd474bd28dcc9bdc15cc1f253166688f: Status 404 returned error can't find the container with id 85a234e207dac2b93d9497d4f1ea8a98dd474bd28dcc9bdc15cc1f253166688f Sep 19 10:39:47.626157 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:47.626125 2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Sep 19 10:39:47.626318 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:47.626176 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" podUID="9026a8a4-9b07-4752-ab91-5c331a6fbeba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Sep 19 10:39:47.739944 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:47.739868 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" event={"ID":"9dd3d0c6-e1f0-4381-acfa-535576a57096","Type":"ContainerStarted","Data":"85a234e207dac2b93d9497d4f1ea8a98dd474bd28dcc9bdc15cc1f253166688f"} Sep 19 10:39:48.626253 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:48.626204 2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Sep 19 10:39:48.626432 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:48.626292 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" podUID="9026a8a4-9b07-4752-ab91-5c331a6fbeba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Sep 19 10:39:48.745725 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:48.745651 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" podUID="9026a8a4-9b07-4752-ab91-5c331a6fbeba" containerName="istio-proxy" containerID="cri-o://afa4e75f542aa1d21cc2922379137d27d76cf7150d548c10df60990313661083" gracePeriod=30 Sep 19 10:39:50.697942 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:50.697908 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c7d4469d9-wpkp5" Sep 19 10:39:50.759766 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:50.759731 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" event={"ID":"9dd3d0c6-e1f0-4381-acfa-535576a57096","Type":"ContainerStarted","Data":"4404b35be71567fb96fed1b5cb05fdfa40f23a72c1ccfa8e997374ea1eea09d9"} Sep 19 10:39:50.759921 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:50.759821 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" Sep 19 10:39:53.001959 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.001894 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" podStartSLOduration=4.471993822 podStartE2EDuration="7.001873307s" totalImagesPullingTime="2.529879485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:39:50.775306991 +0000 UTC m=+322.687811709" watchObservedRunningTime="2026-09-19 10:39:53.001873307 +0000 UTC m=+324.914378024" Sep 19 10:39:53.003536 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.003517 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7"] Sep 19 10:39:53.006994 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.006973 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.010149 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.010122 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 19 10:39:53.010149 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.010136 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jj8ff\"" Sep 19 10:39:53.010149 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.010139 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 19 10:39:53.010427 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.010124 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 19 10:39:53.021110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.021091 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7"] Sep 19 10:39:53.152523 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.152489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade9616b-a22e-4326-8cf8-1fb942ae6084-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.152523 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.152527 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade9616b-a22e-4326-8cf8-1fb942ae6084-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.152732 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.152547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ade9616b-a22e-4326-8cf8-1fb942ae6084-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.152732 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.152629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfxlg\" (UniqueName: \"kubernetes.io/projected/ade9616b-a22e-4326-8cf8-1fb942ae6084-kube-api-access-cfxlg\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.152732 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.152687 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ade9616b-a22e-4326-8cf8-1fb942ae6084-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.152834 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.152754 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ade9616b-a22e-4326-8cf8-1fb942ae6084-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.152834 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.152774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ade9616b-a22e-4326-8cf8-1fb942ae6084-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.254018 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.253928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ade9616b-a22e-4326-8cf8-1fb942ae6084-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.254018 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.253970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ade9616b-a22e-4326-8cf8-1fb942ae6084-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.254018 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.254006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade9616b-a22e-4326-8cf8-1fb942ae6084-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.254338 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.254034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade9616b-a22e-4326-8cf8-1fb942ae6084-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.254338 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.254051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ade9616b-a22e-4326-8cf8-1fb942ae6084-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.254338 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.254144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxlg\" (UniqueName: \"kubernetes.io/projected/ade9616b-a22e-4326-8cf8-1fb942ae6084-kube-api-access-cfxlg\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.254338 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.254170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ade9616b-a22e-4326-8cf8-1fb942ae6084-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.254873 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.254848 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ade9616b-a22e-4326-8cf8-1fb942ae6084-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.256482 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.256459 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade9616b-a22e-4326-8cf8-1fb942ae6084-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.256577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.256501 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ade9616b-a22e-4326-8cf8-1fb942ae6084-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.256814 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.256794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ade9616b-a22e-4326-8cf8-1fb942ae6084-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.256882 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.256827 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ade9616b-a22e-4326-8cf8-1fb942ae6084-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.262553 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.262531 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade9616b-a22e-4326-8cf8-1fb942ae6084-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.262644 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.262561 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfxlg\" (UniqueName: \"kubernetes.io/projected/ade9616b-a22e-4326-8cf8-1fb942ae6084-kube-api-access-cfxlg\") pod \"istiod-openshift-gateway-5677ccb49b-mjlb7\" (UID: \"ade9616b-a22e-4326-8cf8-1fb942ae6084\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.316930 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.316908 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:53.610050 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.610025 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7"] Sep 19 10:39:53.612024 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:39:53.611995 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade9616b_a22e_4326_8cf8_1fb942ae6084.slice/crio-f96b6c5e8d8be4e7de64606cfc92050db7370ea219eb2c0fb1f0ce23e743a971 WatchSource:0}: Error finding container f96b6c5e8d8be4e7de64606cfc92050db7370ea219eb2c0fb1f0ce23e743a971: Status 404 returned error can't find the container with id f96b6c5e8d8be4e7de64606cfc92050db7370ea219eb2c0fb1f0ce23e743a971 Sep 19 10:39:53.779358 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.779326 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" event={"ID":"ade9616b-a22e-4326-8cf8-1fb942ae6084","Type":"ContainerStarted","Data":"f96b6c5e8d8be4e7de64606cfc92050db7370ea219eb2c0fb1f0ce23e743a971"} Sep 19 10:39:53.981749 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:53.981723 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:54.162614 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.162577 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istiod-ca-cert\") pod \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " Sep 19 10:39:54.163110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.162650 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-workload-certs\") pod \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " Sep 19 10:39:54.163110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.162678 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2965\" (UniqueName: \"kubernetes.io/projected/9026a8a4-9b07-4752-ab91-5c331a6fbeba-kube-api-access-w2965\") pod \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " Sep 19 10:39:54.163110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.162717 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-workload-socket\") pod \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " Sep 19 10:39:54.163110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.162750 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-podinfo\") pod \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " Sep 19 10:39:54.163110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.162827 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-credential-socket\") pod \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " Sep 19 10:39:54.163110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.162851 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-envoy\") pod \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " Sep 19 10:39:54.163110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.162886 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-data\") pod \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " Sep 19 10:39:54.163110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.162977 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9026a8a4-9b07-4752-ab91-5c331a6fbeba" (UID: "9026a8a4-9b07-4752-ab91-5c331a6fbeba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:39:54.163110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.163044 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9026a8a4-9b07-4752-ab91-5c331a6fbeba" (UID: "9026a8a4-9b07-4752-ab91-5c331a6fbeba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:39:54.163536 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.163139 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9026a8a4-9b07-4752-ab91-5c331a6fbeba" (UID: "9026a8a4-9b07-4752-ab91-5c331a6fbeba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:39:54.163536 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.163150 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9026a8a4-9b07-4752-ab91-5c331a6fbeba" (UID: "9026a8a4-9b07-4752-ab91-5c331a6fbeba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:39:54.163536 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.163252 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-data" (OuterVolumeSpecName: "istio-data") pod "9026a8a4-9b07-4752-ab91-5c331a6fbeba" (UID: "9026a8a4-9b07-4752-ab91-5c331a6fbeba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:39:54.163536 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.163319 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-token\") pod \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\" (UID: \"9026a8a4-9b07-4752-ab91-5c331a6fbeba\") " Sep 19 10:39:54.163683 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.163635 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:39:54.163683 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.163654 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:39:54.163744 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.163683 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:39:54.163744 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.163698 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:39:54.163744 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.163711 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:39:54.165144 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.165124 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9026a8a4-9b07-4752-ab91-5c331a6fbeba" (UID: "9026a8a4-9b07-4752-ab91-5c331a6fbeba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 10:39:54.165199 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.165138 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9026a8a4-9b07-4752-ab91-5c331a6fbeba-kube-api-access-w2965" (OuterVolumeSpecName: "kube-api-access-w2965") pod "9026a8a4-9b07-4752-ab91-5c331a6fbeba" (UID: "9026a8a4-9b07-4752-ab91-5c331a6fbeba"). InnerVolumeSpecName "kube-api-access-w2965". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:39:54.165404 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.165383 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-token" (OuterVolumeSpecName: "istio-token") pod "9026a8a4-9b07-4752-ab91-5c331a6fbeba" (UID: "9026a8a4-9b07-4752-ab91-5c331a6fbeba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:39:54.165475 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.165380 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9026a8a4-9b07-4752-ab91-5c331a6fbeba" (UID: "9026a8a4-9b07-4752-ab91-5c331a6fbeba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:39:54.264707 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.264678 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:39:54.264707 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.264705 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:39:54.264868 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.264714 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2965\" (UniqueName: \"kubernetes.io/projected/9026a8a4-9b07-4752-ab91-5c331a6fbeba-kube-api-access-w2965\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:39:54.264868 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.264724 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9026a8a4-9b07-4752-ab91-5c331a6fbeba-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:39:54.786365 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.786324 2579 generic.go:356] "Generic (PLEG): container finished" podID="9026a8a4-9b07-4752-ab91-5c331a6fbeba" containerID="afa4e75f542aa1d21cc2922379137d27d76cf7150d548c10df60990313661083" exitCode=0 Sep 19 10:39:54.786564 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.786386 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" Sep 19 10:39:54.786564 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.786382 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" event={"ID":"9026a8a4-9b07-4752-ab91-5c331a6fbeba","Type":"ContainerDied","Data":"afa4e75f542aa1d21cc2922379137d27d76cf7150d548c10df60990313661083"} Sep 19 10:39:54.786564 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.786529 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6" event={"ID":"9026a8a4-9b07-4752-ab91-5c331a6fbeba","Type":"ContainerDied","Data":"35bb394762bdf280417e87879bf4203a5b9e2a6b55f6e0d4ae3a871f261d8f5f"} Sep 19 10:39:54.786564 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.786560 2579 scope.go:117] "RemoveContainer" containerID="afa4e75f542aa1d21cc2922379137d27d76cf7150d548c10df60990313661083" Sep 19 10:39:54.803852 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.803831 2579 scope.go:117] "RemoveContainer" containerID="afa4e75f542aa1d21cc2922379137d27d76cf7150d548c10df60990313661083" Sep 19 10:39:54.804223 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:39:54.804201 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa4e75f542aa1d21cc2922379137d27d76cf7150d548c10df60990313661083\": container with ID starting with afa4e75f542aa1d21cc2922379137d27d76cf7150d548c10df60990313661083 not found: ID does not exist" containerID="afa4e75f542aa1d21cc2922379137d27d76cf7150d548c10df60990313661083" Sep 19 10:39:54.804305 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.804232 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa4e75f542aa1d21cc2922379137d27d76cf7150d548c10df60990313661083"} err="failed to get container status \"afa4e75f542aa1d21cc2922379137d27d76cf7150d548c10df60990313661083\": rpc error: code = NotFound desc = could not find container \"afa4e75f542aa1d21cc2922379137d27d76cf7150d548c10df60990313661083\": container with ID starting with afa4e75f542aa1d21cc2922379137d27d76cf7150d548c10df60990313661083 not found: ID does not exist" Sep 19 10:39:54.831358 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.831322 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6"] Sep 19 10:39:54.835446 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:54.835419 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6"] Sep 19 10:39:56.082795 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:56.082754 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:39:56.083085 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:56.082836 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:39:56.197641 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:56.197605 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9026a8a4-9b07-4752-ab91-5c331a6fbeba" path="/var/lib/kubelet/pods/9026a8a4-9b07-4752-ab91-5c331a6fbeba/volumes" Sep 19 10:39:56.808018 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:56.807978 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" event={"ID":"ade9616b-a22e-4326-8cf8-1fb942ae6084","Type":"ContainerStarted","Data":"0c16eab482d3aca5d498ed25b7244040977f401c81f1b166db049d7bf12bbdf7"} Sep 19 10:39:56.808239 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:56.808055 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:39:56.834833 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:56.834787 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" podStartSLOduration=2.366467976 podStartE2EDuration="4.834770342s" totalImagesPullingTime="2.468302366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:39:56.833157466 +0000 UTC m=+328.745662205" watchObservedRunningTime="2026-09-19 10:39:56.834770342 +0000 UTC m=+328.747275059" Sep 19 10:39:57.813997 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:39:57.813969 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7" Sep 19 10:40:01.766028 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:40:01.765996 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-m7hz5" Sep 19 10:41:24.231916 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.231882 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv"] Sep 19 10:41:24.232366 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.232224 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9026a8a4-9b07-4752-ab91-5c331a6fbeba" containerName="istio-proxy" Sep 19 10:41:24.232366 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.232235 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9026a8a4-9b07-4752-ab91-5c331a6fbeba" containerName="istio-proxy" Sep 19 10:41:24.232366 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.232321 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9026a8a4-9b07-4752-ab91-5c331a6fbeba" containerName="istio-proxy" Sep 19 10:41:24.235248 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.235232 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" Sep 19 10:41:24.237828 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.237806 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-smhjc\"" Sep 19 10:41:24.238120 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.238105 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 19 10:41:24.238174 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.238130 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 19 10:41:24.249691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.249666 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv"] Sep 19 10:41:24.309394 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.309372 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/462e5c82-85a0-4144-94bb-4ed3682f10f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-58svv\" (UID: \"462e5c82-85a0-4144-94bb-4ed3682f10f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" Sep 19 10:41:24.309524 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.309420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqqq2\" (UniqueName: \"kubernetes.io/projected/462e5c82-85a0-4144-94bb-4ed3682f10f2-kube-api-access-vqqq2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-58svv\" (UID: \"462e5c82-85a0-4144-94bb-4ed3682f10f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" Sep 19 10:41:24.409840 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.409815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/462e5c82-85a0-4144-94bb-4ed3682f10f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-58svv\" (UID: \"462e5c82-85a0-4144-94bb-4ed3682f10f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" Sep 19 10:41:24.409944 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.409859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqq2\" (UniqueName: \"kubernetes.io/projected/462e5c82-85a0-4144-94bb-4ed3682f10f2-kube-api-access-vqqq2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-58svv\" (UID: \"462e5c82-85a0-4144-94bb-4ed3682f10f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" Sep 19 10:41:24.410188 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.410169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/462e5c82-85a0-4144-94bb-4ed3682f10f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-58svv\" (UID: \"462e5c82-85a0-4144-94bb-4ed3682f10f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" Sep 19 10:41:24.419054 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.419036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqq2\" (UniqueName: \"kubernetes.io/projected/462e5c82-85a0-4144-94bb-4ed3682f10f2-kube-api-access-vqqq2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-58svv\" (UID: \"462e5c82-85a0-4144-94bb-4ed3682f10f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" Sep 19 10:41:24.545585 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.545521 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" Sep 19 10:41:24.847943 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:24.847919 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv"] Sep 19 10:41:24.850130 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:41:24.850105 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462e5c82_85a0_4144_94bb_4ed3682f10f2.slice/crio-ce694643ce3ce271f16d321dc3c8dc4c4dcd5d5fad3fe1b88cc794413ec8032c WatchSource:0}: Error finding container ce694643ce3ce271f16d321dc3c8dc4c4dcd5d5fad3fe1b88cc794413ec8032c: Status 404 returned error can't find the container with id ce694643ce3ce271f16d321dc3c8dc4c4dcd5d5fad3fe1b88cc794413ec8032c Sep 19 10:41:25.198315 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:25.198283 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" event={"ID":"462e5c82-85a0-4144-94bb-4ed3682f10f2","Type":"ContainerStarted","Data":"ce694643ce3ce271f16d321dc3c8dc4c4dcd5d5fad3fe1b88cc794413ec8032c"} Sep 19 10:41:29.218991 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:29.218958 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" event={"ID":"462e5c82-85a0-4144-94bb-4ed3682f10f2","Type":"ContainerStarted","Data":"dc52d2e760cf2f242f9a36c61e8b25531219fc118aafd1229afc62452006717d"} Sep 19 10:41:29.219467 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:29.219094 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" Sep 19 10:41:29.247299 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:29.247247 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" podStartSLOduration=1.595458187 podStartE2EDuration="5.247229395s" totalImagesPullingTime="3.651771208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:41:29.244275228 +0000 UTC m=+421.156779969" watchObservedRunningTime="2026-09-19 10:41:29.247229395 +0000 UTC m=+421.159734112" Sep 19 10:41:40.228119 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:41:40.228089 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" Sep 19 10:43:19.103639 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.103604 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4nssc"] Sep 19 10:43:19.106915 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.106898 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" Sep 19 10:43:19.109624 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.109601 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pm2vd\"" Sep 19 10:43:19.113786 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.113766 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 19 10:43:19.119560 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.119540 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4nssc"] Sep 19 10:43:19.243220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.243182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcdp\" (UniqueName: \"kubernetes.io/projected/4c0f2be5-2b39-4840-8e24-ea1eb19ba220-kube-api-access-mzcdp\") pod \"payload-processing-5b54b7fbb7-4nssc\" (UID: \"4c0f2be5-2b39-4840-8e24-ea1eb19ba220\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" Sep 19 10:43:19.243220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.243225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0f2be5-2b39-4840-8e24-ea1eb19ba220-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4nssc\" (UID: \"4c0f2be5-2b39-4840-8e24-ea1eb19ba220\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" Sep 19 10:43:19.344597 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.344568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcdp\" (UniqueName: \"kubernetes.io/projected/4c0f2be5-2b39-4840-8e24-ea1eb19ba220-kube-api-access-mzcdp\") pod \"payload-processing-5b54b7fbb7-4nssc\" (UID: \"4c0f2be5-2b39-4840-8e24-ea1eb19ba220\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" Sep 19 10:43:19.344771 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.344605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0f2be5-2b39-4840-8e24-ea1eb19ba220-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4nssc\" (UID: \"4c0f2be5-2b39-4840-8e24-ea1eb19ba220\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" Sep 19 10:43:19.345277 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.345257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0f2be5-2b39-4840-8e24-ea1eb19ba220-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4nssc\" (UID: \"4c0f2be5-2b39-4840-8e24-ea1eb19ba220\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" Sep 19 10:43:19.355758 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.355707 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcdp\" (UniqueName: \"kubernetes.io/projected/4c0f2be5-2b39-4840-8e24-ea1eb19ba220-kube-api-access-mzcdp\") pod \"payload-processing-5b54b7fbb7-4nssc\" (UID: \"4c0f2be5-2b39-4840-8e24-ea1eb19ba220\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" Sep 19 10:43:19.417268 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.417242 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" Sep 19 10:43:19.691665 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.691639 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4nssc"] Sep 19 10:43:19.693545 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:43:19.693517 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0f2be5_2b39_4840_8e24_ea1eb19ba220.slice/crio-af6f3a02f32714dd6fc1feed366e6b9328f7e3901471737bec82f6eb4490e926 WatchSource:0}: Error finding container af6f3a02f32714dd6fc1feed366e6b9328f7e3901471737bec82f6eb4490e926: Status 404 returned error can't find the container with id af6f3a02f32714dd6fc1feed366e6b9328f7e3901471737bec82f6eb4490e926 Sep 19 10:43:19.718001 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:19.717977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" event={"ID":"4c0f2be5-2b39-4840-8e24-ea1eb19ba220","Type":"ContainerStarted","Data":"af6f3a02f32714dd6fc1feed366e6b9328f7e3901471737bec82f6eb4490e926"} Sep 19 10:43:22.736471 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:22.736437 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" event={"ID":"4c0f2be5-2b39-4840-8e24-ea1eb19ba220","Type":"ContainerStarted","Data":"bb2055910a6c8f8763750e1f6b23bea02bc289e7e8c9bd0bff2cb450715e38e2"} Sep 19 10:43:22.736848 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:22.736572 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" Sep 19 10:43:22.755450 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:22.755404 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" podStartSLOduration=1.455948789 podStartE2EDuration="3.75539052s" totalImagesPullingTime="2.299441731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:43:22.753364811 +0000 UTC m=+534.665869528" watchObservedRunningTime="2026-09-19 10:43:22.75539052 +0000 UTC m=+534.667895236" Sep 19 10:43:33.743479 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:43:33.743449 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" Sep 19 10:44:15.346595 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.346556 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj"] Sep 19 10:44:15.350025 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.350006 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.353675 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.353655 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4rnbf\"" Sep 19 10:44:15.353799 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.353677 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 19 10:44:15.353799 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.353724 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 19 10:44:15.353799 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.353771 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 19 10:44:15.360356 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.360336 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj"] Sep 19 10:44:15.404750 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.404721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de0ea4e-a064-4046-a797-f691acd64344-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.404898 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.404757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de0ea4e-a064-4046-a797-f691acd64344-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.404898 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.404790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1de0ea4e-a064-4046-a797-f691acd64344-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.404898 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.404876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4cz2\" (UniqueName: \"kubernetes.io/projected/1de0ea4e-a064-4046-a797-f691acd64344-kube-api-access-w4cz2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.405086 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.404951 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1de0ea4e-a064-4046-a797-f691acd64344-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.405086 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.405049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1de0ea4e-a064-4046-a797-f691acd64344-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.506473 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.506444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1de0ea4e-a064-4046-a797-f691acd64344-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.506659 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.506481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1de0ea4e-a064-4046-a797-f691acd64344-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.506659 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.506511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de0ea4e-a064-4046-a797-f691acd64344-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.506659 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.506625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de0ea4e-a064-4046-a797-f691acd64344-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.506812 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.506680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1de0ea4e-a064-4046-a797-f691acd64344-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.506812 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.506713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4cz2\" (UniqueName: \"kubernetes.io/projected/1de0ea4e-a064-4046-a797-f691acd64344-kube-api-access-w4cz2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.506939 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.506908 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1de0ea4e-a064-4046-a797-f691acd64344-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.507001 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.506958 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1de0ea4e-a064-4046-a797-f691acd64344-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.507043 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.507016 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de0ea4e-a064-4046-a797-f691acd64344-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.508792 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.508772 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1de0ea4e-a064-4046-a797-f691acd64344-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.508938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.508920 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de0ea4e-a064-4046-a797-f691acd64344-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.515004 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.514981 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4cz2\" (UniqueName: \"kubernetes.io/projected/1de0ea4e-a064-4046-a797-f691acd64344-kube-api-access-w4cz2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9fhrj\" (UID: \"1de0ea4e-a064-4046-a797-f691acd64344\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.661224 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.661187 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:15.935352 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.935325 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj"] Sep 19 10:44:15.936900 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:44:15.936874 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de0ea4e_a064_4046_a797_f691acd64344.slice/crio-c7cf53ff14f4b9d124d6eb2e85bf6ed07e370fa67fea8517f6c8dcbe0ac96071 WatchSource:0}: Error finding container c7cf53ff14f4b9d124d6eb2e85bf6ed07e370fa67fea8517f6c8dcbe0ac96071: Status 404 returned error can't find the container with id c7cf53ff14f4b9d124d6eb2e85bf6ed07e370fa67fea8517f6c8dcbe0ac96071 Sep 19 10:44:15.984917 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:15.984887 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" event={"ID":"1de0ea4e-a064-4046-a797-f691acd64344","Type":"ContainerStarted","Data":"c7cf53ff14f4b9d124d6eb2e85bf6ed07e370fa67fea8517f6c8dcbe0ac96071"} Sep 19 10:44:16.991041 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:16.991009 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" event={"ID":"1de0ea4e-a064-4046-a797-f691acd64344","Type":"ContainerStarted","Data":"fa3d4b88e590582fd3017d6ae7a20919c96bf53be134d59ce36f33effeb4cd68"} Sep 19 10:44:17.013309 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:17.013267 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" podStartSLOduration=1.071554281 podStartE2EDuration="2.013252438s" totalImagesPullingTime="941.698157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:44:17.011515683 +0000 UTC m=+588.924020428" watchObservedRunningTime="2026-09-19 10:44:17.013252438 +0000 UTC m=+588.925757153" Sep 19 10:44:25.661268 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:25.661237 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:25.661677 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:25.661295 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:25.673744 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:25.673720 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:26.044517 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:26.044444 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9fhrj" Sep 19 10:44:30.168684 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:30.168658 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 10:44:30.170798 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:30.170776 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 10:44:30.874083 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:30.874039 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx"] Sep 19 10:44:30.877789 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:30.877769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:30.880921 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:30.880901 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 19 10:44:30.893554 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:30.893530 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx"] Sep 19 10:44:30.934027 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:30.933990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5407dfe0-508b-464b-8c4f-fab89f076018-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:30.934148 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:30.934034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5407dfe0-508b-464b-8c4f-fab89f076018-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:30.934148 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:30.934079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6v67\" (UniqueName: \"kubernetes.io/projected/5407dfe0-508b-464b-8c4f-fab89f076018-kube-api-access-t6v67\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:30.934227 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:30.934192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5407dfe0-508b-464b-8c4f-fab89f076018-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:30.934262 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:30.934231 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5407dfe0-508b-464b-8c4f-fab89f076018-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:30.934309 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:30.934293 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5407dfe0-508b-464b-8c4f-fab89f076018-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.035442 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.035410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5407dfe0-508b-464b-8c4f-fab89f076018-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.035442 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.035446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5407dfe0-508b-464b-8c4f-fab89f076018-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.035659 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.035471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6v67\" (UniqueName: \"kubernetes.io/projected/5407dfe0-508b-464b-8c4f-fab89f076018-kube-api-access-t6v67\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.035659 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.035532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5407dfe0-508b-464b-8c4f-fab89f076018-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.035659 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.035554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5407dfe0-508b-464b-8c4f-fab89f076018-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.035659 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.035592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5407dfe0-508b-464b-8c4f-fab89f076018-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.035913 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.035819 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5407dfe0-508b-464b-8c4f-fab89f076018-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.036003 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.035983 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5407dfe0-508b-464b-8c4f-fab89f076018-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.036082 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.036036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5407dfe0-508b-464b-8c4f-fab89f076018-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.037841 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.037820 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5407dfe0-508b-464b-8c4f-fab89f076018-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.037981 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.037964 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5407dfe0-508b-464b-8c4f-fab89f076018-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.043794 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.043772 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6v67\" (UniqueName: \"kubernetes.io/projected/5407dfe0-508b-464b-8c4f-fab89f076018-kube-api-access-t6v67\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2nzkx\" (UID: \"5407dfe0-508b-464b-8c4f-fab89f076018\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.188613 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.188579 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:31.459679 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:31.459654 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx"] Sep 19 10:44:31.461818 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:44:31.461785 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5407dfe0_508b_464b_8c4f_fab89f076018.slice/crio-578a977f951241bb1e0ee9c1372682b8ccc27e457fb3a997e8322d5e2e999b3f WatchSource:0}: Error finding container 578a977f951241bb1e0ee9c1372682b8ccc27e457fb3a997e8322d5e2e999b3f: Status 404 returned error can't find the container with id 578a977f951241bb1e0ee9c1372682b8ccc27e457fb3a997e8322d5e2e999b3f Sep 19 10:44:32.064317 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:32.064278 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" event={"ID":"5407dfe0-508b-464b-8c4f-fab89f076018","Type":"ContainerStarted","Data":"f79b57971e7fe5030680063af095509f49d4fd0f10b8b134680c90e29f2dbf40"} Sep 19 10:44:32.064483 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:32.064345 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" event={"ID":"5407dfe0-508b-464b-8c4f-fab89f076018","Type":"ContainerStarted","Data":"578a977f951241bb1e0ee9c1372682b8ccc27e457fb3a997e8322d5e2e999b3f"} Sep 19 10:44:32.084995 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:32.084951 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" podStartSLOduration=1.910986549 podStartE2EDuration="2.084937089s" totalImagesPullingTime="173.95054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:44:32.083458164 +0000 UTC m=+603.995962879" watchObservedRunningTime="2026-09-19 10:44:32.084937089 +0000 UTC m=+603.997441805" Sep 19 10:44:41.189526 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:41.189485 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:41.192547 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:41.189594 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:41.201701 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:41.201683 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:44:42.128907 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:44:42.128881 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2nzkx" Sep 19 10:46:36.343164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.343123 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4nssc"] Sep 19 10:46:36.343629 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.343365 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" podUID="4c0f2be5-2b39-4840-8e24-ea1eb19ba220" containerName="payload-processing" containerID="cri-o://bb2055910a6c8f8763750e1f6b23bea02bc289e7e8c9bd0bff2cb450715e38e2" gracePeriod=30 Sep 19 10:46:36.594176 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.594119 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" Sep 19 10:46:36.681160 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.681128 2579 generic.go:356] "Generic (PLEG): container finished" podID="4c0f2be5-2b39-4840-8e24-ea1eb19ba220" containerID="bb2055910a6c8f8763750e1f6b23bea02bc289e7e8c9bd0bff2cb450715e38e2" exitCode=0 Sep 19 10:46:36.681322 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.681168 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" Sep 19 10:46:36.681322 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.681213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" event={"ID":"4c0f2be5-2b39-4840-8e24-ea1eb19ba220","Type":"ContainerDied","Data":"bb2055910a6c8f8763750e1f6b23bea02bc289e7e8c9bd0bff2cb450715e38e2"} Sep 19 10:46:36.681322 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.681263 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nssc" event={"ID":"4c0f2be5-2b39-4840-8e24-ea1eb19ba220","Type":"ContainerDied","Data":"af6f3a02f32714dd6fc1feed366e6b9328f7e3901471737bec82f6eb4490e926"} Sep 19 10:46:36.681322 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.681287 2579 scope.go:117] "RemoveContainer" containerID="bb2055910a6c8f8763750e1f6b23bea02bc289e7e8c9bd0bff2cb450715e38e2" Sep 19 10:46:36.693109 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.693079 2579 scope.go:117] "RemoveContainer" containerID="bb2055910a6c8f8763750e1f6b23bea02bc289e7e8c9bd0bff2cb450715e38e2" Sep 19 10:46:36.693625 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:46:36.693599 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2055910a6c8f8763750e1f6b23bea02bc289e7e8c9bd0bff2cb450715e38e2\": container with ID starting with bb2055910a6c8f8763750e1f6b23bea02bc289e7e8c9bd0bff2cb450715e38e2 not found: ID does not exist" containerID="bb2055910a6c8f8763750e1f6b23bea02bc289e7e8c9bd0bff2cb450715e38e2" Sep 19 10:46:36.693731 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.693634 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2055910a6c8f8763750e1f6b23bea02bc289e7e8c9bd0bff2cb450715e38e2"} err="failed to get container status \"bb2055910a6c8f8763750e1f6b23bea02bc289e7e8c9bd0bff2cb450715e38e2\": rpc error: code = NotFound desc = could not find container \"bb2055910a6c8f8763750e1f6b23bea02bc289e7e8c9bd0bff2cb450715e38e2\": container with ID starting with bb2055910a6c8f8763750e1f6b23bea02bc289e7e8c9bd0bff2cb450715e38e2 not found: ID does not exist" Sep 19 10:46:36.709900 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.709876 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzcdp\" (UniqueName: \"kubernetes.io/projected/4c0f2be5-2b39-4840-8e24-ea1eb19ba220-kube-api-access-mzcdp\") pod \"4c0f2be5-2b39-4840-8e24-ea1eb19ba220\" (UID: \"4c0f2be5-2b39-4840-8e24-ea1eb19ba220\") " Sep 19 10:46:36.710079 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.710048 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0f2be5-2b39-4840-8e24-ea1eb19ba220-plugins-config-volume\") pod \"4c0f2be5-2b39-4840-8e24-ea1eb19ba220\" (UID: \"4c0f2be5-2b39-4840-8e24-ea1eb19ba220\") " Sep 19 10:46:36.710379 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.710356 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0f2be5-2b39-4840-8e24-ea1eb19ba220-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c0f2be5-2b39-4840-8e24-ea1eb19ba220" (UID: "4c0f2be5-2b39-4840-8e24-ea1eb19ba220"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:46:36.711901 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.711877 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0f2be5-2b39-4840-8e24-ea1eb19ba220-kube-api-access-mzcdp" (OuterVolumeSpecName: "kube-api-access-mzcdp") pod "4c0f2be5-2b39-4840-8e24-ea1eb19ba220" (UID: "4c0f2be5-2b39-4840-8e24-ea1eb19ba220"). InnerVolumeSpecName "kube-api-access-mzcdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:46:36.811534 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.811505 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzcdp\" (UniqueName: \"kubernetes.io/projected/4c0f2be5-2b39-4840-8e24-ea1eb19ba220-kube-api-access-mzcdp\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:46:36.811534 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:36.811530 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0f2be5-2b39-4840-8e24-ea1eb19ba220-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:46:37.009247 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:37.006611 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4nssc"] Sep 19 10:46:37.010863 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:37.010838 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4nssc"] Sep 19 10:46:38.195876 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:38.195842 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c0f2be5-2b39-4840-8e24-ea1eb19ba220" path="/var/lib/kubelet/pods/4c0f2be5-2b39-4840-8e24-ea1eb19ba220/volumes" Sep 19 10:46:54.331353 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.331319 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p"] Sep 19 10:46:54.331938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.331762 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c0f2be5-2b39-4840-8e24-ea1eb19ba220" containerName="payload-processing" Sep 19 10:46:54.331938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.331775 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0f2be5-2b39-4840-8e24-ea1eb19ba220" containerName="payload-processing" Sep 19 10:46:54.331938 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.331861 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c0f2be5-2b39-4840-8e24-ea1eb19ba220" containerName="payload-processing" Sep 19 10:46:54.335168 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.335144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p" Sep 19 10:46:54.337583 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.337560 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 19 10:46:54.337701 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.337604 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pm2vd\"" Sep 19 10:46:54.345866 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.345833 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p"] Sep 19 10:46:54.463238 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.463199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324db52d-7f53-4736-a870-200921c89361-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-dkm2p\" (UID: \"324db52d-7f53-4736-a870-200921c89361\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p" Sep 19 10:46:54.463398 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.463293 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bw74\" (UniqueName: \"kubernetes.io/projected/324db52d-7f53-4736-a870-200921c89361-kube-api-access-8bw74\") pod \"payload-pre-processing-76f657cdc4-dkm2p\" (UID: \"324db52d-7f53-4736-a870-200921c89361\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p" Sep 19 10:46:54.563957 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.563925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324db52d-7f53-4736-a870-200921c89361-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-dkm2p\" (UID: \"324db52d-7f53-4736-a870-200921c89361\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p" Sep 19 10:46:54.564148 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.563977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bw74\" (UniqueName: \"kubernetes.io/projected/324db52d-7f53-4736-a870-200921c89361-kube-api-access-8bw74\") pod \"payload-pre-processing-76f657cdc4-dkm2p\" (UID: \"324db52d-7f53-4736-a870-200921c89361\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p" Sep 19 10:46:54.564774 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.564751 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324db52d-7f53-4736-a870-200921c89361-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-dkm2p\" (UID: \"324db52d-7f53-4736-a870-200921c89361\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p" Sep 19 10:46:54.571486 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.571464 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bw74\" (UniqueName: \"kubernetes.io/projected/324db52d-7f53-4736-a870-200921c89361-kube-api-access-8bw74\") pod \"payload-pre-processing-76f657cdc4-dkm2p\" (UID: \"324db52d-7f53-4736-a870-200921c89361\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p" Sep 19 10:46:54.649382 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.649356 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p" Sep 19 10:46:54.917907 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.917882 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p"] Sep 19 10:46:54.919245 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:46:54.919219 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324db52d_7f53_4736_a870_200921c89361.slice/crio-b7cfa26753209365cd73e2bbfe55ba8240faa056891b5963185ad2870b7d8ca0 WatchSource:0}: Error finding container b7cfa26753209365cd73e2bbfe55ba8240faa056891b5963185ad2870b7d8ca0: Status 404 returned error can't find the container with id b7cfa26753209365cd73e2bbfe55ba8240faa056891b5963185ad2870b7d8ca0 Sep 19 10:46:54.921127 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:54.921110 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 19 10:46:55.774581 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:55.774541 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p" event={"ID":"324db52d-7f53-4736-a870-200921c89361","Type":"ContainerStarted","Data":"b7cfa26753209365cd73e2bbfe55ba8240faa056891b5963185ad2870b7d8ca0"} Sep 19 10:46:57.786309 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:57.786273 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p" event={"ID":"324db52d-7f53-4736-a870-200921c89361","Type":"ContainerStarted","Data":"5b60f928d83529742e8d7c6c737c83742c39c6551422e7fec98f48d7a035f6a6"} Sep 19 10:46:57.786768 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:57.786413 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p" Sep 19 10:46:57.801249 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:46:57.801203 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p" podStartSLOduration=1.7216354059999999 podStartE2EDuration="3.801190203s" totalImagesPullingTime="2.079554797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:46:57.800693755 +0000 UTC m=+749.713198475" watchObservedRunningTime="2026-09-19 10:46:57.801190203 +0000 UTC m=+749.713694919" Sep 19 10:47:08.793255 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:08.793213 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p" Sep 19 10:47:30.106473 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.106443 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt"] Sep 19 10:47:30.123960 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.123927 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.126532 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.126501 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-580e01-openshift-default-dockercfg-qmmbs\"" Sep 19 10:47:30.127885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.127860 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt"] Sep 19 10:47:30.171559 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.171524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdp8d\" (UniqueName: \"kubernetes.io/projected/6939ddf4-8535-4764-ac63-12646e8a14bb-kube-api-access-mdp8d\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.171708 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.171575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-workload-certs\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.171708 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.171672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-credential-socket\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.171810 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.171711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-podinfo\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.171810 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.171746 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-data\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.171975 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.171833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-token\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.171975 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.171892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-workload-socket\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.171975 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.171935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-envoy\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.172140 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.171984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6939ddf4-8535-4764-ac63-12646e8a14bb-istiod-ca-cert\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.173430 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.173407 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8"] Sep 19 10:47:30.177842 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.177825 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.181619 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.181598 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f91f97-openshift-default-dockercfg-7zp8g\"" Sep 19 10:47:30.198030 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.198004 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8"] Sep 19 10:47:30.273276 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdp8d\" (UniqueName: \"kubernetes.io/projected/6939ddf4-8535-4764-ac63-12646e8a14bb-kube-api-access-mdp8d\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.273733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-token\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.273733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-envoy\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.273733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-istio-data\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.273733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273429 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a8f190d-f536-49af-b687-28adddb7733f-istio-token\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.273733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273484 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-istio-envoy\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.273733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6939ddf4-8535-4764-ac63-12646e8a14bb-istiod-ca-cert\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.273733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273551 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-workload-socket\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.273733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-workload-certs\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.273733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a8f190d-f536-49af-b687-28adddb7733f-istio-podinfo\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.273733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a8f190d-f536-49af-b687-28adddb7733f-istiod-ca-cert\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.273733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-credential-socket\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.273733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-credential-socket\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.273733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-podinfo\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.274441 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-workload-certs\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.274441 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-workload-socket\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.274441 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-data\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.274441 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.273936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97rjs\" (UniqueName: \"kubernetes.io/projected/4a8f190d-f536-49af-b687-28adddb7733f-kube-api-access-97rjs\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.274441 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.274287 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-data\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.274707 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.274614 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6939ddf4-8535-4764-ac63-12646e8a14bb-istiod-ca-cert\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.274935 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.274907 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-credential-socket\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.275253 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.275228 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-workload-certs\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.275487 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.275465 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-workload-socket\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.276833 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.276808 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-envoy\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.277237 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.277212 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-podinfo\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.287587 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.287554 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-token\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.288801 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.288589 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdp8d\" (UniqueName: \"kubernetes.io/projected/6939ddf4-8535-4764-ac63-12646e8a14bb-kube-api-access-mdp8d\") pod \"e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.374535 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.374461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97rjs\" (UniqueName: \"kubernetes.io/projected/4a8f190d-f536-49af-b687-28adddb7733f-kube-api-access-97rjs\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.374699 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.374536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-istio-data\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.374699 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.374574 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a8f190d-f536-49af-b687-28adddb7733f-istio-token\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.374699 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.374687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-istio-envoy\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.374871 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.374734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-workload-socket\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.374871 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.374775 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a8f190d-f536-49af-b687-28adddb7733f-istio-podinfo\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.374871 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.374817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a8f190d-f536-49af-b687-28adddb7733f-istiod-ca-cert\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.374871 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.374847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-credential-socket\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.375094 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.374872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-workload-certs\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.375094 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.374963 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-istio-data\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.375094 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.375017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-workload-socket\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.375258 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.375193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-credential-socket\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.375313 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.375253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-workload-certs\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.375664 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.375640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a8f190d-f536-49af-b687-28adddb7733f-istiod-ca-cert\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.377508 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.377482 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-istio-envoy\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.377746 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.377725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a8f190d-f536-49af-b687-28adddb7733f-istio-podinfo\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.390852 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.390825 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a8f190d-f536-49af-b687-28adddb7733f-istio-token\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.390934 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.390924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97rjs\" (UniqueName: \"kubernetes.io/projected/4a8f190d-f536-49af-b687-28adddb7733f-kube-api-access-97rjs\") pod \"e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.437052 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.437030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:30.489132 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.489085 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:30.711999 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.711975 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt"] Sep 19 10:47:30.712682 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:47:30.712662 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6939ddf4_8535_4764_ac63_12646e8a14bb.slice/crio-04ed59e988abd97c79412e1550cd2994c31b809bde4564bee3fe62990a0f7fc1 WatchSource:0}: Error finding container 04ed59e988abd97c79412e1550cd2994c31b809bde4564bee3fe62990a0f7fc1: Status 404 returned error can't find the container with id 04ed59e988abd97c79412e1550cd2994c31b809bde4564bee3fe62990a0f7fc1 Sep 19 10:47:30.714827 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.714797 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:30.714884 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.714858 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:30.714920 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.714883 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:30.771233 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.771208 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8"] Sep 19 10:47:30.774014 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:47:30.773985 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8f190d_f536_49af_b687_28adddb7733f.slice/crio-c4d0b675e9163c4882c640e4f569a0f8cc39e07decb2e671ee250b46506f2006 WatchSource:0}: Error finding container c4d0b675e9163c4882c640e4f569a0f8cc39e07decb2e671ee250b46506f2006: Status 404 returned error can't find the container with id c4d0b675e9163c4882c640e4f569a0f8cc39e07decb2e671ee250b46506f2006 Sep 19 10:47:30.776294 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.776267 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:30.776349 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.776335 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:30.776382 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.776362 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:30.943107 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.943040 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" event={"ID":"4a8f190d-f536-49af-b687-28adddb7733f","Type":"ContainerStarted","Data":"2d0b4fe430b814274cedbf17ee414369dda787a91b4538446cb9ccec3ee428c2"} Sep 19 10:47:30.943280 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.943133 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" event={"ID":"4a8f190d-f536-49af-b687-28adddb7733f","Type":"ContainerStarted","Data":"c4d0b675e9163c4882c640e4f569a0f8cc39e07decb2e671ee250b46506f2006"} Sep 19 10:47:30.945553 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.945518 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" event={"ID":"6939ddf4-8535-4764-ac63-12646e8a14bb","Type":"ContainerStarted","Data":"e2ed3960a1cd134414eae7dc1de2e3e911481d79bff8f085da28ff3680839b0e"} Sep 19 10:47:30.945691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.945571 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" event={"ID":"6939ddf4-8535-4764-ac63-12646e8a14bb","Type":"ContainerStarted","Data":"04ed59e988abd97c79412e1550cd2994c31b809bde4564bee3fe62990a0f7fc1"} Sep 19 10:47:30.964824 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:30.964732 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" podStartSLOduration=0.964712835 podStartE2EDuration="964.712835ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:47:30.962330916 +0000 UTC m=+782.874835636" watchObservedRunningTime="2026-09-19 10:47:30.964712835 +0000 UTC m=+782.877217551" Sep 19 10:47:31.437599 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:31.437568 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:31.442468 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:31.442443 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:31.461947 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:31.461901 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" podStartSLOduration=1.461884259 podStartE2EDuration="1.461884259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:47:30.98129659 +0000 UTC m=+782.893801307" watchObservedRunningTime="2026-09-19 10:47:31.461884259 +0000 UTC m=+783.374389053" Sep 19 10:47:31.489951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:31.489927 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:31.494730 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:31.494707 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:31.952427 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:31.952378 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:31.952615 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:31.952447 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:31.953173 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:31.953149 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:31.953276 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:31.953210 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:34.770832 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:34.770796 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb"] Sep 19 10:47:34.774801 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:34.774780 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" Sep 19 10:47:34.779249 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:34.779229 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f91f97-dockercfg-qxmnc\"" Sep 19 10:47:34.779536 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:34.779293 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f91f97\"" Sep 19 10:47:34.787315 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:34.787294 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb"] Sep 19 10:47:34.921895 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:34.921857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-465sp\" (UniqueName: \"kubernetes.io/projected/1d2dd5b8-8563-4db8-b1e0-6fedc133fbee-kube-api-access-465sp\") pod \"payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb\" (UID: \"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" Sep 19 10:47:34.922117 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:34.921999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2dd5b8-8563-4db8-b1e0-6fedc133fbee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb\" (UID: \"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" Sep 19 10:47:35.023633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.023530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2dd5b8-8563-4db8-b1e0-6fedc133fbee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb\" (UID: \"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" Sep 19 10:47:35.023633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.023597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-465sp\" (UniqueName: \"kubernetes.io/projected/1d2dd5b8-8563-4db8-b1e0-6fedc133fbee-kube-api-access-465sp\") pod \"payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb\" (UID: \"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" Sep 19 10:47:35.024302 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.024279 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2dd5b8-8563-4db8-b1e0-6fedc133fbee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb\" (UID: \"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" Sep 19 10:47:35.031591 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.031568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-465sp\" (UniqueName: \"kubernetes.io/projected/1d2dd5b8-8563-4db8-b1e0-6fedc133fbee-kube-api-access-465sp\") pod \"payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb\" (UID: \"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" Sep 19 10:47:35.086400 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.086370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" Sep 19 10:47:35.359269 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.359241 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb"] Sep 19 10:47:35.361973 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:47:35.361946 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2dd5b8_8563_4db8_b1e0_6fedc133fbee.slice/crio-451ebd2752bc4073448614fe01a4db81d49d694f648424c4321480b47fd36ffc WatchSource:0}: Error finding container 451ebd2752bc4073448614fe01a4db81d49d694f648424c4321480b47fd36ffc: Status 404 returned error can't find the container with id 451ebd2752bc4073448614fe01a4db81d49d694f648424c4321480b47fd36ffc Sep 19 10:47:35.791744 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.791711 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk"] Sep 19 10:47:35.795436 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.795418 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" Sep 19 10:47:35.797768 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.797748 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-580e01\"" Sep 19 10:47:35.797909 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.797788 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-580e01-dockercfg-4fkpq\"" Sep 19 10:47:35.805146 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.805124 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk"] Sep 19 10:47:35.835894 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.835871 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv"] Sep 19 10:47:35.839379 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.839361 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" Sep 19 10:47:35.854363 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.854340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv"] Sep 19 10:47:35.931856 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.931827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bcec4da-5b43-48e3-bfed-33511225d966-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv\" (UID: \"8bcec4da-5b43-48e3-bfed-33511225d966\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" Sep 19 10:47:35.932035 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.931899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbl4f\" (UniqueName: \"kubernetes.io/projected/8bcec4da-5b43-48e3-bfed-33511225d966-kube-api-access-vbl4f\") pod \"payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv\" (UID: \"8bcec4da-5b43-48e3-bfed-33511225d966\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" Sep 19 10:47:35.932035 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.931922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz75k\" (UniqueName: \"kubernetes.io/projected/0d62ccc5-db7b-4638-bbf1-4e7297c0f68a-kube-api-access-nz75k\") pod \"payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk\" (UID: \"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" Sep 19 10:47:35.932035 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.931939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d62ccc5-db7b-4638-bbf1-4e7297c0f68a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk\" (UID: \"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" Sep 19 10:47:35.975550 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.975518 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" event={"ID":"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee","Type":"ContainerStarted","Data":"196eb7a2b4f079ae556673d046ec6932c105bf0e217dd4c2d64cc553abd81744"} Sep 19 10:47:35.975733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.975575 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" event={"ID":"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee","Type":"ContainerStarted","Data":"451ebd2752bc4073448614fe01a4db81d49d694f648424c4321480b47fd36ffc"} Sep 19 10:47:35.975733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.975642 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" Sep 19 10:47:35.994370 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:35.994325 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" podStartSLOduration=1.994309473 podStartE2EDuration="1.994309473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:47:35.993131627 +0000 UTC m=+787.905636344" watchObservedRunningTime="2026-09-19 10:47:35.994309473 +0000 UTC m=+787.906814220" Sep 19 10:47:36.033304 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.033276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bcec4da-5b43-48e3-bfed-33511225d966-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv\" (UID: \"8bcec4da-5b43-48e3-bfed-33511225d966\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" Sep 19 10:47:36.033461 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.033353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbl4f\" (UniqueName: \"kubernetes.io/projected/8bcec4da-5b43-48e3-bfed-33511225d966-kube-api-access-vbl4f\") pod \"payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv\" (UID: \"8bcec4da-5b43-48e3-bfed-33511225d966\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" Sep 19 10:47:36.033461 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.033373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz75k\" (UniqueName: \"kubernetes.io/projected/0d62ccc5-db7b-4638-bbf1-4e7297c0f68a-kube-api-access-nz75k\") pod \"payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk\" (UID: \"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" Sep 19 10:47:36.033461 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.033420 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d62ccc5-db7b-4638-bbf1-4e7297c0f68a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk\" (UID: \"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" Sep 19 10:47:36.034152 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.034124 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bcec4da-5b43-48e3-bfed-33511225d966-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv\" (UID: \"8bcec4da-5b43-48e3-bfed-33511225d966\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" Sep 19 10:47:36.034246 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.034155 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d62ccc5-db7b-4638-bbf1-4e7297c0f68a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk\" (UID: \"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" Sep 19 10:47:36.041711 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.041680 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz75k\" (UniqueName: \"kubernetes.io/projected/0d62ccc5-db7b-4638-bbf1-4e7297c0f68a-kube-api-access-nz75k\") pod \"payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk\" (UID: \"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" Sep 19 10:47:36.041985 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.041921 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbl4f\" (UniqueName: \"kubernetes.io/projected/8bcec4da-5b43-48e3-bfed-33511225d966-kube-api-access-vbl4f\") pod \"payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv\" (UID: \"8bcec4da-5b43-48e3-bfed-33511225d966\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" Sep 19 10:47:36.107475 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.107100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" Sep 19 10:47:36.151402 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.149687 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" Sep 19 10:47:36.387874 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.387774 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk"] Sep 19 10:47:36.389820 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:47:36.389797 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d62ccc5_db7b_4638_bbf1_4e7297c0f68a.slice/crio-2aeb48f56bc5b0f931698352f4312e12b7b7a77a42a0635f0701c4c57cc12762 WatchSource:0}: Error finding container 2aeb48f56bc5b0f931698352f4312e12b7b7a77a42a0635f0701c4c57cc12762: Status 404 returned error can't find the container with id 2aeb48f56bc5b0f931698352f4312e12b7b7a77a42a0635f0701c4c57cc12762 Sep 19 10:47:36.454810 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.454706 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv"] Sep 19 10:47:36.459275 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:47:36.459249 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcec4da_5b43_48e3_bfed_33511225d966.slice/crio-4cf2f3d47e1b0234d2033a21d477a85d819b885f6b66f00d92c5863a8989b079 WatchSource:0}: Error finding container 4cf2f3d47e1b0234d2033a21d477a85d819b885f6b66f00d92c5863a8989b079: Status 404 returned error can't find the container with id 4cf2f3d47e1b0234d2033a21d477a85d819b885f6b66f00d92c5863a8989b079 Sep 19 10:47:36.982358 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.982319 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" event={"ID":"8bcec4da-5b43-48e3-bfed-33511225d966","Type":"ContainerStarted","Data":"99f975e75903d302bbd6b905a138d0cf52c0050b014a6c1fec1c6213c49bd9d5"} Sep 19 10:47:36.982845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.982384 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" event={"ID":"8bcec4da-5b43-48e3-bfed-33511225d966","Type":"ContainerStarted","Data":"4cf2f3d47e1b0234d2033a21d477a85d819b885f6b66f00d92c5863a8989b079"} Sep 19 10:47:36.982845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.982597 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" Sep 19 10:47:36.984094 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.984053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" event={"ID":"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a","Type":"ContainerStarted","Data":"75426e78207cfbc848158df6d0d695928cab3522aa326e2f752712e97db4e75f"} Sep 19 10:47:36.984223 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.984122 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" event={"ID":"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a","Type":"ContainerStarted","Data":"2aeb48f56bc5b0f931698352f4312e12b7b7a77a42a0635f0701c4c57cc12762"} Sep 19 10:47:36.984223 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:36.984176 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" Sep 19 10:47:37.002669 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:37.002620 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" podStartSLOduration=2.002603742 podStartE2EDuration="2.002603742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:47:37.00017294 +0000 UTC m=+788.912677691" watchObservedRunningTime="2026-09-19 10:47:37.002603742 +0000 UTC m=+788.915108457" Sep 19 10:47:37.015518 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:37.015474 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" podStartSLOduration=2.015458381 podStartE2EDuration="2.015458381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:47:37.014774886 +0000 UTC m=+788.927279604" watchObservedRunningTime="2026-09-19 10:47:37.015458381 +0000 UTC m=+788.927963097" Sep 19 10:47:38.796023 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.795993 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr"] Sep 19 10:47:38.799826 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.799800 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.802312 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.802293 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b21bfa16-openshift-default-dockercfg-x5p55\"" Sep 19 10:47:38.813650 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.813625 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr"] Sep 19 10:47:38.864972 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.864941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-envoy\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.865148 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.864993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkmnt\" (UniqueName: \"kubernetes.io/projected/5074cb52-5e88-4450-89d4-bef1f809a1e3-kube-api-access-fkmnt\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.865148 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.865025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-token\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.865148 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.865132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-workload-certs\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.865338 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.865174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5074cb52-5e88-4450-89d4-bef1f809a1e3-istiod-ca-cert\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.865338 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.865282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-data\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.865442 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.865335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-podinfo\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.865442 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.865378 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-workload-socket\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.865442 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.865420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-credential-socket\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.966266 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.966222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-credential-socket\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.966418 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.966287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-envoy\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.966418 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.966323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkmnt\" (UniqueName: \"kubernetes.io/projected/5074cb52-5e88-4450-89d4-bef1f809a1e3-kube-api-access-fkmnt\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.966418 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.966349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-token\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.966418 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.966411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-workload-certs\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.966640 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.966449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5074cb52-5e88-4450-89d4-bef1f809a1e3-istiod-ca-cert\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.966640 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.966496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-data\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.966640 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.966533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-podinfo\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.966640 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.966590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-workload-socket\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.967243 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.967220 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-workload-certs\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.967413 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.967351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-workload-socket\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.967504 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.967400 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5074cb52-5e88-4450-89d4-bef1f809a1e3-istiod-ca-cert\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.967612 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.967542 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-credential-socket\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.967612 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.967545 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-data\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.969126 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.969100 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-envoy\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.969468 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.969449 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-podinfo\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.975312 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.975289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-token\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:38.975770 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:38.975753 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkmnt\" (UniqueName: \"kubernetes.io/projected/5074cb52-5e88-4450-89d4-bef1f809a1e3-kube-api-access-fkmnt\") pod \"e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:39.112011 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:39.111935 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:39.388042 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:39.388014 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr"] Sep 19 10:47:39.389888 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:47:39.389857 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5074cb52_5e88_4450_89d4_bef1f809a1e3.slice/crio-7efa61492e0ede21c1aa26e17f5caac32b09be95e2a1efa0b88301a0e4d6aa19 WatchSource:0}: Error finding container 7efa61492e0ede21c1aa26e17f5caac32b09be95e2a1efa0b88301a0e4d6aa19: Status 404 returned error can't find the container with id 7efa61492e0ede21c1aa26e17f5caac32b09be95e2a1efa0b88301a0e4d6aa19 Sep 19 10:47:39.392054 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:39.392025 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:39.392179 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:39.392114 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:39.392179 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:39.392146 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:40.002369 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:40.002338 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" event={"ID":"5074cb52-5e88-4450-89d4-bef1f809a1e3","Type":"ContainerStarted","Data":"42dfedcc6869ef18cf27d63234a9280efb3b1f4f674dc2a0e6d48c164250059b"} Sep 19 10:47:40.002759 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:40.002386 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" event={"ID":"5074cb52-5e88-4450-89d4-bef1f809a1e3","Type":"ContainerStarted","Data":"7efa61492e0ede21c1aa26e17f5caac32b09be95e2a1efa0b88301a0e4d6aa19"} Sep 19 10:47:40.024916 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:40.024869 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" podStartSLOduration=2.02485346 podStartE2EDuration="2.02485346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:47:40.023105381 +0000 UTC m=+791.935610096" watchObservedRunningTime="2026-09-19 10:47:40.02485346 +0000 UTC m=+791.937358175" Sep 19 10:47:40.112158 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:40.112113 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:40.117158 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:40.117134 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:41.007967 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:41.007935 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:41.008758 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:41.008741 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:47:44.866655 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:44.861606 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr"] Sep 19 10:47:44.866655 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:44.866025 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" Sep 19 10:47:44.875978 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:44.875560 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b21bfa16\"" Sep 19 10:47:44.875978 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:44.875809 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b21bfa16-dockercfg-gwrsn\"" Sep 19 10:47:44.893727 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:44.893699 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr"] Sep 19 10:47:45.028951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:45.028914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd4mj\" (UniqueName: \"kubernetes.io/projected/18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9-kube-api-access-dd4mj\") pod \"payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr\" (UID: \"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9\") " pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" Sep 19 10:47:45.028951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:45.028953 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9-plugins-config-volume\") pod \"payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr\" (UID: \"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9\") " pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" Sep 19 10:47:45.130281 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:45.130205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd4mj\" (UniqueName: \"kubernetes.io/projected/18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9-kube-api-access-dd4mj\") pod \"payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr\" (UID: \"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9\") " pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" Sep 19 10:47:45.130281 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:45.130236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9-plugins-config-volume\") pod \"payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr\" (UID: \"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9\") " pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" Sep 19 10:47:45.130964 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:45.130943 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9-plugins-config-volume\") pod \"payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr\" (UID: \"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9\") " pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" Sep 19 10:47:45.137685 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:45.137658 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd4mj\" (UniqueName: \"kubernetes.io/projected/18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9-kube-api-access-dd4mj\") pod \"payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr\" (UID: \"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9\") " pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" Sep 19 10:47:45.198804 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:45.198762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" Sep 19 10:47:45.513600 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:45.513574 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr"] Sep 19 10:47:45.515410 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:47:45.515383 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ffb1c1_7fa6_4659_9a3b_9ee01ffdddc9.slice/crio-cd82ab4cd8b183a101974d07d6acaf9926aa630faee49914d0af4d887928baac WatchSource:0}: Error finding container cd82ab4cd8b183a101974d07d6acaf9926aa630faee49914d0af4d887928baac: Status 404 returned error can't find the container with id cd82ab4cd8b183a101974d07d6acaf9926aa630faee49914d0af4d887928baac Sep 19 10:47:46.036032 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:46.035994 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" event={"ID":"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9","Type":"ContainerStarted","Data":"dd4c5581ac92411b1b8741a5f2598e507d785e2ee4c2d010058e7c10b7078090"} Sep 19 10:47:46.036404 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:46.036079 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" event={"ID":"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9","Type":"ContainerStarted","Data":"cd82ab4cd8b183a101974d07d6acaf9926aa630faee49914d0af4d887928baac"} Sep 19 10:47:46.036404 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:46.036248 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" Sep 19 10:47:46.060872 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:46.060813 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" podStartSLOduration=2.060790009 podStartE2EDuration="2.060790009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:47:46.057109545 +0000 UTC m=+797.969614262" watchObservedRunningTime="2026-09-19 10:47:46.060790009 +0000 UTC m=+797.973294726" Sep 19 10:47:46.986702 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:46.986597 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" Sep 19 10:47:47.992643 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:47.992496 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" Sep 19 10:47:47.992643 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:47.992593 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" Sep 19 10:47:47.994222 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:47.994192 2579 patch_prober.go:28] interesting pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 19 10:47:47.994337 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:47.994248 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 19 10:47:48.290217 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.289982 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr"] Sep 19 10:47:48.294500 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.294480 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.297321 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.297084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6cf6a0-openshift-default-dockercfg-bvvc6\"" Sep 19 10:47:48.307803 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.307768 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr"] Sep 19 10:47:48.368956 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.368921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-credential-socket\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.368956 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.368966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd05d1e-9759-421e-826f-44441269f943-istiod-ca-cert\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.369223 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.369034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-workload-socket\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.369223 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.369146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd05d1e-9759-421e-826f-44441269f943-istio-podinfo\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.369223 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.369208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9p8\" (UniqueName: \"kubernetes.io/projected/ddd05d1e-9759-421e-826f-44441269f943-kube-api-access-5w9p8\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.369362 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.369229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-istio-envoy\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.369362 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.369278 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-istio-data\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.369362 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.369346 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd05d1e-9759-421e-826f-44441269f943-istio-token\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.369517 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.369385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-workload-certs\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.470273 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.470236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9p8\" (UniqueName: \"kubernetes.io/projected/ddd05d1e-9759-421e-826f-44441269f943-kube-api-access-5w9p8\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.470273 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.470279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-istio-envoy\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.470512 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.470326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-istio-data\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.470512 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.470359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd05d1e-9759-421e-826f-44441269f943-istio-token\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.470512 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.470395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-workload-certs\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.470512 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.470424 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-credential-socket\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.470512 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.470451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd05d1e-9759-421e-826f-44441269f943-istiod-ca-cert\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.470512 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.470498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-workload-socket\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.470837 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.470565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd05d1e-9759-421e-826f-44441269f943-istio-podinfo\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.470837 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.470796 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-istio-data\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.471340 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.470974 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-credential-socket\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.471340 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.471265 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-workload-certs\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.471510 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.471383 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-workload-socket\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.471707 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.471684 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd05d1e-9759-421e-826f-44441269f943-istiod-ca-cert\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.472878 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.472843 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-istio-envoy\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.473231 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.473214 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd05d1e-9759-421e-826f-44441269f943-istio-podinfo\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.478915 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.478894 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9p8\" (UniqueName: \"kubernetes.io/projected/ddd05d1e-9759-421e-826f-44441269f943-kube-api-access-5w9p8\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.480054 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.480027 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd05d1e-9759-421e-826f-44441269f943-istio-token\") pod \"e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.620241 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.620145 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:48.904558 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.904516 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr"] Sep 19 10:47:48.906846 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:47:48.906816 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd05d1e_9759_421e_826f_44441269f943.slice/crio-170c35f98987a8f9e2b8e4e5227a0bd08fc90b3209b607e8eca38e7e9cd8a744 WatchSource:0}: Error finding container 170c35f98987a8f9e2b8e4e5227a0bd08fc90b3209b607e8eca38e7e9cd8a744: Status 404 returned error can't find the container with id 170c35f98987a8f9e2b8e4e5227a0bd08fc90b3209b607e8eca38e7e9cd8a744 Sep 19 10:47:48.909722 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.909691 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:48.909823 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.909767 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:48.909823 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:48.909808 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:49.061129 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:49.061091 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" event={"ID":"ddd05d1e-9759-421e-826f-44441269f943","Type":"ContainerStarted","Data":"ab252c1636a12eda2a494db82bf5f31ad7390766aad67927a563908b73d60e9b"} Sep 19 10:47:49.061484 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:49.061156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" event={"ID":"ddd05d1e-9759-421e-826f-44441269f943","Type":"ContainerStarted","Data":"170c35f98987a8f9e2b8e4e5227a0bd08fc90b3209b607e8eca38e7e9cd8a744"} Sep 19 10:47:49.081027 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:49.080970 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" podStartSLOduration=1.080948614 podStartE2EDuration="1.080948614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:47:49.078815947 +0000 UTC m=+800.991320665" watchObservedRunningTime="2026-09-19 10:47:49.080948614 +0000 UTC m=+800.993453331" Sep 19 10:47:49.620947 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:49.620911 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:49.627193 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:49.627159 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:50.067646 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:50.067621 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:50.126590 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:50.068748 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:47:51.074268 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:51.074237 2579 generic.go:356] "Generic (PLEG): container finished" podID="4a8f190d-f536-49af-b687-28adddb7733f" containerID="2d0b4fe430b814274cedbf17ee414369dda787a91b4538446cb9ccec3ee428c2" exitCode=0 Sep 19 10:47:51.074735 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:51.074304 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" event={"ID":"4a8f190d-f536-49af-b687-28adddb7733f","Type":"ContainerDied","Data":"2d0b4fe430b814274cedbf17ee414369dda787a91b4538446cb9ccec3ee428c2"} Sep 19 10:47:51.074833 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:51.074809 2579 scope.go:117] "RemoveContainer" containerID="2d0b4fe430b814274cedbf17ee414369dda787a91b4538446cb9ccec3ee428c2" Sep 19 10:47:51.075990 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:51.075963 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:51.076130 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:51.075989 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" event={"ID":"6939ddf4-8535-4764-ac63-12646e8a14bb","Type":"ContainerDied","Data":"e2ed3960a1cd134414eae7dc1de2e3e911481d79bff8f085da28ff3680839b0e"} Sep 19 10:47:51.076130 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:51.076051 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:51.076130 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:51.076119 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:51.076287 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:51.075968 2579 generic.go:356] "Generic (PLEG): container finished" podID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerID="e2ed3960a1cd134414eae7dc1de2e3e911481d79bff8f085da28ff3680839b0e" exitCode=0 Sep 19 10:47:51.077839 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:51.076926 2579 scope.go:117] "RemoveContainer" containerID="e2ed3960a1cd134414eae7dc1de2e3e911481d79bff8f085da28ff3680839b0e" Sep 19 10:47:51.082246 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:51.079693 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:51.082246 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:51.080327 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:51.082246 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:51.080486 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:47:52.106564 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:52.106460 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" event={"ID":"4a8f190d-f536-49af-b687-28adddb7733f","Type":"ContainerStarted","Data":"575e52a79f1f70337b626cf28f6869f08a8bc46d6d5b8e4b849822cc274e5ae2"} Sep 19 10:47:52.109239 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:52.109207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" event={"ID":"6939ddf4-8535-4764-ac63-12646e8a14bb","Type":"ContainerStarted","Data":"a8e85408b73a35af91294e8b838e5af718b9e177dabd6d5e9304d867acd39cc4"} Sep 19 10:47:52.437610 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:52.437582 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:52.437833 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:52.437743 2579 patch_prober.go:28] interesting pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 19 10:47:52.437949 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:52.437933 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 19 10:47:52.490231 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:52.490170 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:52.490450 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:52.490416 2579 patch_prober.go:28] interesting pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 19 10:47:52.490522 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:52.490490 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 19 10:47:53.438022 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:53.437987 2579 patch_prober.go:28] interesting pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 19 10:47:53.438571 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:53.438040 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 19 10:47:53.489740 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:53.489701 2579 patch_prober.go:28] interesting pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 19 10:47:53.490008 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:53.489978 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 19 10:47:54.437814 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:54.437778 2579 patch_prober.go:28] interesting pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 19 10:47:54.438004 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:54.437831 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 19 10:47:54.491081 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:54.490613 2579 patch_prober.go:28] interesting pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 19 10:47:54.491081 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:54.490690 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 19 10:47:55.437700 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:55.437665 2579 patch_prober.go:28] interesting pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 19 10:47:55.437900 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:55.437722 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 19 10:47:55.489832 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:55.489798 2579 patch_prober.go:28] interesting pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 19 10:47:55.490158 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:55.490106 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 19 10:47:56.437653 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:56.437605 2579 patch_prober.go:28] interesting pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 19 10:47:56.438165 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:56.437672 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 19 10:47:56.490082 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:56.489720 2579 patch_prober.go:28] interesting pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 19 10:47:56.490082 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:56.489779 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 19 10:47:56.580245 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:56.580180 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v"] Sep 19 10:47:56.586314 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:56.586287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v" Sep 19 10:47:56.589531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:56.589509 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 10:47:56.589830 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:56.589539 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 10:47:56.601597 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:56.600164 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v"] Sep 19 10:47:56.672083 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:56.671106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d7beac-7e7c-46c1-9362-c777992313af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v\" (UID: \"e3d7beac-7e7c-46c1-9362-c777992313af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v" Sep 19 10:47:56.772588 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:56.772473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d7beac-7e7c-46c1-9362-c777992313af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v\" (UID: \"e3d7beac-7e7c-46c1-9362-c777992313af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v" Sep 19 10:47:56.773508 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:56.773475 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d7beac-7e7c-46c1-9362-c777992313af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v\" (UID: \"e3d7beac-7e7c-46c1-9362-c777992313af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v" Sep 19 10:47:56.902790 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:56.902732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v" Sep 19 10:47:57.070083 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:57.063045 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" Sep 19 10:47:57.288273 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:57.279699 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v"] Sep 19 10:47:57.437884 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:57.437845 2579 patch_prober.go:28] interesting pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 19 10:47:57.438337 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:57.437905 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 19 10:47:57.504091 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:57.503178 2579 patch_prober.go:28] interesting pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 19 10:47:57.504091 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:57.503245 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 19 10:47:58.162735 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:58.162704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v" event={"ID":"e3d7beac-7e7c-46c1-9362-c777992313af","Type":"ContainerStarted","Data":"1a445dd026c8bba3934c715cea047e397b750b2eacc481349ca4e8e49e57df5c"} Sep 19 10:47:58.442667 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:58.442574 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:58.443320 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:58.443292 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:58.444207 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:58.444182 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:47:58.497093 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:58.496544 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:58.498021 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:58.497958 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:47:58.499600 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:47:58.499499 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:48:00.177124 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:00.177087 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v" event={"ID":"e3d7beac-7e7c-46c1-9362-c777992313af","Type":"ContainerStarted","Data":"64b6995452fbeb9407b55a60bb45e30aca7c4d42bf2366f9d0062b6da004dfd5"} Sep 19 10:48:00.192290 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:00.192239 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v" podStartSLOduration=1.8317469530000001 podStartE2EDuration="4.192219634s" totalImagesPullingTime="2.360472681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:48:00.190882578 +0000 UTC m=+812.103387296" watchObservedRunningTime="2026-09-19 10:48:00.192219634 +0000 UTC m=+812.104724352" Sep 19 10:48:01.183816 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:01.183776 2579 generic.go:356] "Generic (PLEG): container finished" podID="e3d7beac-7e7c-46c1-9362-c777992313af" containerID="64b6995452fbeb9407b55a60bb45e30aca7c4d42bf2366f9d0062b6da004dfd5" exitCode=0 Sep 19 10:48:01.183816 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:01.183858 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v" event={"ID":"e3d7beac-7e7c-46c1-9362-c777992313af","Type":"ContainerDied","Data":"64b6995452fbeb9407b55a60bb45e30aca7c4d42bf2366f9d0062b6da004dfd5"} Sep 19 10:48:02.190860 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:02.190823 2579 generic.go:356] "Generic (PLEG): container finished" podID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerID="42dfedcc6869ef18cf27d63234a9280efb3b1f4f674dc2a0e6d48c164250059b" exitCode=0 Sep 19 10:48:02.196609 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:02.196582 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" event={"ID":"5074cb52-5e88-4450-89d4-bef1f809a1e3","Type":"ContainerDied","Data":"42dfedcc6869ef18cf27d63234a9280efb3b1f4f674dc2a0e6d48c164250059b"} Sep 19 10:48:02.197004 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:02.196986 2579 scope.go:117] "RemoveContainer" containerID="42dfedcc6869ef18cf27d63234a9280efb3b1f4f674dc2a0e6d48c164250059b" Sep 19 10:48:02.198108 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:02.198082 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:02.198211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:02.198163 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:02.198275 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:02.198213 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:02.346252 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:02.346226 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v" Sep 19 10:48:02.454245 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:02.454152 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d7beac-7e7c-46c1-9362-c777992313af-maas-api-service-ca\") pod \"e3d7beac-7e7c-46c1-9362-c777992313af\" (UID: \"e3d7beac-7e7c-46c1-9362-c777992313af\") " Sep 19 10:48:02.454493 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:02.454468 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d7beac-7e7c-46c1-9362-c777992313af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e3d7beac-7e7c-46c1-9362-c777992313af" (UID: "e3d7beac-7e7c-46c1-9362-c777992313af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:48:02.454747 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:02.454725 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d7beac-7e7c-46c1-9362-c777992313af-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:03.198495 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:03.198464 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" event={"ID":"5074cb52-5e88-4450-89d4-bef1f809a1e3","Type":"ContainerStarted","Data":"9f2137be2e64fff3c25f065874723b0549aefc64c0f9bac716ae3fd130dfeb14"} Sep 19 10:48:03.199763 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:03.199738 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v" event={"ID":"e3d7beac-7e7c-46c1-9362-c777992313af","Type":"ContainerDied","Data":"1a445dd026c8bba3934c715cea047e397b750b2eacc481349ca4e8e49e57df5c"} Sep 19 10:48:03.199763 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:03.199758 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v" Sep 19 10:48:03.199908 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:03.199780 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a445dd026c8bba3934c715cea047e397b750b2eacc481349ca4e8e49e57df5c" Sep 19 10:48:04.112256 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:04.112225 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:48:04.112435 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:04.112402 2579 patch_prober.go:28] interesting pod/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 19 10:48:04.112483 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:04.112449 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" podUID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 19 10:48:05.112361 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:05.112322 2579 patch_prober.go:28] interesting pod/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 19 10:48:05.112759 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:05.112380 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" podUID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 19 10:48:06.112704 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:06.112670 2579 patch_prober.go:28] interesting pod/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 19 10:48:06.113116 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:06.112723 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" podUID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 19 10:48:07.112434 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:07.112395 2579 patch_prober.go:28] interesting pod/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 19 10:48:07.112608 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:07.112449 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" podUID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 19 10:48:08.112934 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.112896 2579 patch_prober.go:28] interesting pod/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 19 10:48:08.113357 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.112950 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" podUID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 19 10:48:08.299179 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.296218 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr"] Sep 19 10:48:08.299179 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.296457 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" podUID="18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9" containerName="payload-processing" containerID="cri-o://dd4c5581ac92411b1b8741a5f2598e507d785e2ee4c2d010058e7c10b7078090" gracePeriod=30 Sep 19 10:48:08.433935 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.433895 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v"] Sep 19 10:48:08.436173 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.436135 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b21bfa16-zbr2v"] Sep 19 10:48:08.573852 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.573820 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr"] Sep 19 10:48:08.577955 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.574378 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" podUID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerName="istio-proxy" containerID="cri-o://9f2137be2e64fff3c25f065874723b0549aefc64c0f9bac716ae3fd130dfeb14" gracePeriod=30 Sep 19 10:48:08.714223 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.714192 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" Sep 19 10:48:08.823821 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.823787 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9-plugins-config-volume\") pod \"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9\" (UID: \"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9\") " Sep 19 10:48:08.824015 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.823876 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd4mj\" (UniqueName: \"kubernetes.io/projected/18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9-kube-api-access-dd4mj\") pod \"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9\" (UID: \"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9\") " Sep 19 10:48:08.824207 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.824182 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9" (UID: "18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:48:08.826447 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.826417 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9-kube-api-access-dd4mj" (OuterVolumeSpecName: "kube-api-access-dd4mj") pod "18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9" (UID: "18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9"). InnerVolumeSpecName "kube-api-access-dd4mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:48:08.924890 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.924859 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:08.924890 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:08.924887 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd4mj\" (UniqueName: \"kubernetes.io/projected/18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9-kube-api-access-dd4mj\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:09.234074 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.234019 2579 generic.go:356] "Generic (PLEG): container finished" podID="18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9" containerID="dd4c5581ac92411b1b8741a5f2598e507d785e2ee4c2d010058e7c10b7078090" exitCode=0 Sep 19 10:48:09.234502 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.234097 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" Sep 19 10:48:09.234502 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.234099 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" event={"ID":"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9","Type":"ContainerDied","Data":"dd4c5581ac92411b1b8741a5f2598e507d785e2ee4c2d010058e7c10b7078090"} Sep 19 10:48:09.234502 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.234159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr" event={"ID":"18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9","Type":"ContainerDied","Data":"cd82ab4cd8b183a101974d07d6acaf9926aa630faee49914d0af4d887928baac"} Sep 19 10:48:09.234502 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.234181 2579 scope.go:117] "RemoveContainer" containerID="dd4c5581ac92411b1b8741a5f2598e507d785e2ee4c2d010058e7c10b7078090" Sep 19 10:48:09.264244 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.264214 2579 scope.go:117] "RemoveContainer" containerID="dd4c5581ac92411b1b8741a5f2598e507d785e2ee4c2d010058e7c10b7078090" Sep 19 10:48:09.264947 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:48:09.264911 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4c5581ac92411b1b8741a5f2598e507d785e2ee4c2d010058e7c10b7078090\": container with ID starting with dd4c5581ac92411b1b8741a5f2598e507d785e2ee4c2d010058e7c10b7078090 not found: ID does not exist" containerID="dd4c5581ac92411b1b8741a5f2598e507d785e2ee4c2d010058e7c10b7078090" Sep 19 10:48:09.265075 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.264957 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4c5581ac92411b1b8741a5f2598e507d785e2ee4c2d010058e7c10b7078090"} err="failed to get container status \"dd4c5581ac92411b1b8741a5f2598e507d785e2ee4c2d010058e7c10b7078090\": rpc error: code = NotFound desc = could not find container \"dd4c5581ac92411b1b8741a5f2598e507d785e2ee4c2d010058e7c10b7078090\": container with ID starting with dd4c5581ac92411b1b8741a5f2598e507d785e2ee4c2d010058e7c10b7078090 not found: ID does not exist" Sep 19 10:48:09.281734 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.281705 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr"] Sep 19 10:48:09.283651 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.283632 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr"] Sep 19 10:48:09.526953 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.526875 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw"] Sep 19 10:48:09.527492 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.527470 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9" containerName="payload-processing" Sep 19 10:48:09.527492 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.527494 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9" containerName="payload-processing" Sep 19 10:48:09.527616 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.527508 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d7beac-7e7c-46c1-9362-c777992313af" containerName="revoke-keys" Sep 19 10:48:09.527616 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.527514 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d7beac-7e7c-46c1-9362-c777992313af" containerName="revoke-keys" Sep 19 10:48:09.527616 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.527590 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d7beac-7e7c-46c1-9362-c777992313af" containerName="revoke-keys" Sep 19 10:48:09.527616 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.527597 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9" containerName="payload-processing" Sep 19 10:48:09.532578 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.532557 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.535553 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.535533 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fb988ee6-gw-openshift-default-dockercfg-dvp9z\"" Sep 19 10:48:09.560601 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.560552 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw"] Sep 19 10:48:09.630975 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.630937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-credential-socket\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.631176 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.631036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-token\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.631176 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.631101 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-podinfo\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.631176 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.631133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-envoy\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.631356 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.631189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dkw\" (UniqueName: \"kubernetes.io/projected/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-kube-api-access-m8dkw\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.631356 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.631231 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istiod-ca-cert\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.631356 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.631249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-workload-socket\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.631356 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.631279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-workload-certs\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.631356 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.631330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-data\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.732302 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.732269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-data\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.732446 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.732329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-credential-socket\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.732446 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.732371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-token\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.732446 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.732423 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-podinfo\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.732594 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.732460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-envoy\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.732594 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.732500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dkw\" (UniqueName: \"kubernetes.io/projected/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-kube-api-access-m8dkw\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.732594 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.732548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istiod-ca-cert\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.732594 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.732577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-workload-socket\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.732823 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.732610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-workload-certs\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.732823 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.732702 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-credential-socket\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.732823 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.732760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-data\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.732953 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.732863 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-workload-certs\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.733018 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.732982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-workload-socket\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.733236 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.733217 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istiod-ca-cert\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.734873 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.734853 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-envoy\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.735005 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.734990 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-podinfo\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.740551 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.740505 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-token\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.742702 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.742674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dkw\" (UniqueName: \"kubernetes.io/projected/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-kube-api-access-m8dkw\") pod \"e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:09.857695 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:09.857595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:10.147590 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:10.147558 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw"] Sep 19 10:48:10.148107 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:48:10.148082 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a52fa2_1f48_4ca4_a6dc_0991744dc7b0.slice/crio-9b34cc0528a49b6421fa2a9dc26beaa27c7dd5c5953746123438b533cbb36842 WatchSource:0}: Error finding container 9b34cc0528a49b6421fa2a9dc26beaa27c7dd5c5953746123438b533cbb36842: Status 404 returned error can't find the container with id 9b34cc0528a49b6421fa2a9dc26beaa27c7dd5c5953746123438b533cbb36842 Sep 19 10:48:10.150639 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:10.150604 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:10.150753 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:10.150680 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:10.150753 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:10.150724 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:10.198023 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:10.197997 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9" path="/var/lib/kubelet/pods/18ffb1c1-7fa6-4659-9a3b-9ee01ffdddc9/volumes" Sep 19 10:48:10.198429 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:10.198412 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d7beac-7e7c-46c1-9362-c777992313af" path="/var/lib/kubelet/pods/e3d7beac-7e7c-46c1-9362-c777992313af/volumes" Sep 19 10:48:10.241544 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:10.241513 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" event={"ID":"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0","Type":"ContainerStarted","Data":"f68e7f7003bcb3548da3a9aa06e760dd8916d82ac8380bfb046856ee2830d95c"} Sep 19 10:48:10.241910 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:10.241575 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" event={"ID":"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0","Type":"ContainerStarted","Data":"9b34cc0528a49b6421fa2a9dc26beaa27c7dd5c5953746123438b533cbb36842"} Sep 19 10:48:10.261698 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:10.261644 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" podStartSLOduration=1.261629256 podStartE2EDuration="1.261629256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:48:10.259287746 +0000 UTC m=+822.171792487" watchObservedRunningTime="2026-09-19 10:48:10.261629256 +0000 UTC m=+822.174133973" Sep 19 10:48:10.858051 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:10.858016 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:11.008817 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:11.008782 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:48:11.863529 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:11.863499 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:12.258568 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:12.258528 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:12.272114 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:12.268336 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:12.790319 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:12.790287 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2"] Sep 19 10:48:12.798630 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:12.798607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" Sep 19 10:48:12.801235 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:12.801210 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fb988ee6-dockercfg-6vlqp\"" Sep 19 10:48:12.801678 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:12.801657 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fb988ee6\"" Sep 19 10:48:12.804480 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:12.804459 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2"] Sep 19 10:48:12.869208 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:12.869160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2\" (UID: \"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" Sep 19 10:48:12.883031 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:12.869328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zg7\" (UniqueName: \"kubernetes.io/projected/ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4-kube-api-access-g9zg7\") pod \"payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2\" (UID: \"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" Sep 19 10:48:12.969928 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:12.969899 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zg7\" (UniqueName: \"kubernetes.io/projected/ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4-kube-api-access-g9zg7\") pod \"payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2\" (UID: \"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" Sep 19 10:48:12.970051 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:12.970013 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2\" (UID: \"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" Sep 19 10:48:12.970656 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:12.970637 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2\" (UID: \"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" Sep 19 10:48:12.977312 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:12.977293 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zg7\" (UniqueName: \"kubernetes.io/projected/ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4-kube-api-access-g9zg7\") pod \"payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2\" (UID: \"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" Sep 19 10:48:13.111267 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:13.111161 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" Sep 19 10:48:13.264688 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:13.264658 2579 generic.go:356] "Generic (PLEG): container finished" podID="ddd05d1e-9759-421e-826f-44441269f943" containerID="ab252c1636a12eda2a494db82bf5f31ad7390766aad67927a563908b73d60e9b" exitCode=0 Sep 19 10:48:13.264834 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:13.264730 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" event={"ID":"ddd05d1e-9759-421e-826f-44441269f943","Type":"ContainerDied","Data":"ab252c1636a12eda2a494db82bf5f31ad7390766aad67927a563908b73d60e9b"} Sep 19 10:48:13.265199 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:13.265186 2579 scope.go:117] "RemoveContainer" containerID="ab252c1636a12eda2a494db82bf5f31ad7390766aad67927a563908b73d60e9b" Sep 19 10:48:13.266245 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:13.266210 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:13.266334 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:13.266287 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:13.266334 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:13.266317 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:13.386034 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:13.385877 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2"] Sep 19 10:48:13.387808 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:48:13.387782 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebcc856d_e6e9_41a0_8972_c8dbcf2fa4b4.slice/crio-8aff1583bdc021c19ca37f671e12e479f0dc0a4d76146b8db308725c08688701 WatchSource:0}: Error finding container 8aff1583bdc021c19ca37f671e12e479f0dc0a4d76146b8db308725c08688701: Status 404 returned error can't find the container with id 8aff1583bdc021c19ca37f671e12e479f0dc0a4d76146b8db308725c08688701 Sep 19 10:48:14.133378 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.133358 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:48:14.282035 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.281995 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" event={"ID":"ddd05d1e-9759-421e-826f-44441269f943","Type":"ContainerStarted","Data":"7bea3617020b32c075cfc2dca898cdf5ee4e492b766d330f926a832e03ceb351"} Sep 19 10:48:14.283640 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.283616 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" event={"ID":"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4","Type":"ContainerStarted","Data":"5ada5da35c1c6b3314a605d7856b0eaca1c3f7d932d8fce8e007476d2802dc6e"} Sep 19 10:48:14.283748 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.283656 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" event={"ID":"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4","Type":"ContainerStarted","Data":"8aff1583bdc021c19ca37f671e12e479f0dc0a4d76146b8db308725c08688701"} Sep 19 10:48:14.283748 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.283711 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" Sep 19 10:48:14.283903 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.283884 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-credential-socket\") pod \"5074cb52-5e88-4450-89d4-bef1f809a1e3\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " Sep 19 10:48:14.283970 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.283927 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-token\") pod \"5074cb52-5e88-4450-89d4-bef1f809a1e3\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " Sep 19 10:48:14.283970 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.283962 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-data\") pod \"5074cb52-5e88-4450-89d4-bef1f809a1e3\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " Sep 19 10:48:14.284093 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284007 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-workload-socket\") pod \"5074cb52-5e88-4450-89d4-bef1f809a1e3\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " Sep 19 10:48:14.284093 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284030 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-workload-certs\") pod \"5074cb52-5e88-4450-89d4-bef1f809a1e3\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " Sep 19 10:48:14.284093 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284084 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkmnt\" (UniqueName: \"kubernetes.io/projected/5074cb52-5e88-4450-89d4-bef1f809a1e3-kube-api-access-fkmnt\") pod \"5074cb52-5e88-4450-89d4-bef1f809a1e3\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " Sep 19 10:48:14.284248 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284152 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5074cb52-5e88-4450-89d4-bef1f809a1e3-istiod-ca-cert\") pod \"5074cb52-5e88-4450-89d4-bef1f809a1e3\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " Sep 19 10:48:14.284248 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284204 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5074cb52-5e88-4450-89d4-bef1f809a1e3" (UID: "5074cb52-5e88-4450-89d4-bef1f809a1e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:48:14.284354 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284215 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-envoy\") pod \"5074cb52-5e88-4450-89d4-bef1f809a1e3\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " Sep 19 10:48:14.284354 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284296 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-podinfo\") pod \"5074cb52-5e88-4450-89d4-bef1f809a1e3\" (UID: \"5074cb52-5e88-4450-89d4-bef1f809a1e3\") " Sep 19 10:48:14.284807 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284549 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5074cb52-5e88-4450-89d4-bef1f809a1e3" (UID: "5074cb52-5e88-4450-89d4-bef1f809a1e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:48:14.284807 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284565 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5074cb52-5e88-4450-89d4-bef1f809a1e3" (UID: "5074cb52-5e88-4450-89d4-bef1f809a1e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:48:14.284807 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284713 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "5074cb52-5e88-4450-89d4-bef1f809a1e3" (UID: "5074cb52-5e88-4450-89d4-bef1f809a1e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:48:14.284807 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284780 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5074cb52-5e88-4450-89d4-bef1f809a1e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5074cb52-5e88-4450-89d4-bef1f809a1e3" (UID: "5074cb52-5e88-4450-89d4-bef1f809a1e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:48:14.285089 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284824 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:14.285089 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284840 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:14.285089 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284854 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:14.285089 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.284872 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:14.285903 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.285881 2579 generic.go:356] "Generic (PLEG): container finished" podID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerID="9f2137be2e64fff3c25f065874723b0549aefc64c0f9bac716ae3fd130dfeb14" exitCode=0 Sep 19 10:48:14.286020 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.286001 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" Sep 19 10:48:14.286104 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.286017 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" event={"ID":"5074cb52-5e88-4450-89d4-bef1f809a1e3","Type":"ContainerDied","Data":"9f2137be2e64fff3c25f065874723b0549aefc64c0f9bac716ae3fd130dfeb14"} Sep 19 10:48:14.286164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.286106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr" event={"ID":"5074cb52-5e88-4450-89d4-bef1f809a1e3","Type":"ContainerDied","Data":"7efa61492e0ede21c1aa26e17f5caac32b09be95e2a1efa0b88301a0e4d6aa19"} Sep 19 10:48:14.286164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.286142 2579 scope.go:117] "RemoveContainer" containerID="9f2137be2e64fff3c25f065874723b0549aefc64c0f9bac716ae3fd130dfeb14" Sep 19 10:48:14.287008 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.286928 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "5074cb52-5e88-4450-89d4-bef1f809a1e3" (UID: "5074cb52-5e88-4450-89d4-bef1f809a1e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:48:14.287629 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.287427 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5074cb52-5e88-4450-89d4-bef1f809a1e3-kube-api-access-fkmnt" (OuterVolumeSpecName: "kube-api-access-fkmnt") pod "5074cb52-5e88-4450-89d4-bef1f809a1e3" (UID: "5074cb52-5e88-4450-89d4-bef1f809a1e3"). InnerVolumeSpecName "kube-api-access-fkmnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:48:14.287629 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.287505 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5074cb52-5e88-4450-89d4-bef1f809a1e3" (UID: "5074cb52-5e88-4450-89d4-bef1f809a1e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 10:48:14.288034 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.287812 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5074cb52-5e88-4450-89d4-bef1f809a1e3" (UID: "5074cb52-5e88-4450-89d4-bef1f809a1e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:48:14.309359 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.309342 2579 scope.go:117] "RemoveContainer" containerID="42dfedcc6869ef18cf27d63234a9280efb3b1f4f674dc2a0e6d48c164250059b" Sep 19 10:48:14.317751 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.317712 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" podStartSLOduration=2.31769687 podStartE2EDuration="2.31769687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:48:14.316239238 +0000 UTC m=+826.228743953" watchObservedRunningTime="2026-09-19 10:48:14.31769687 +0000 UTC m=+826.230201587" Sep 19 10:48:14.319671 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.319653 2579 scope.go:117] "RemoveContainer" containerID="9f2137be2e64fff3c25f065874723b0549aefc64c0f9bac716ae3fd130dfeb14" Sep 19 10:48:14.319965 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:48:14.319947 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2137be2e64fff3c25f065874723b0549aefc64c0f9bac716ae3fd130dfeb14\": container with ID starting with 9f2137be2e64fff3c25f065874723b0549aefc64c0f9bac716ae3fd130dfeb14 not found: ID does not exist" containerID="9f2137be2e64fff3c25f065874723b0549aefc64c0f9bac716ae3fd130dfeb14" Sep 19 10:48:14.320017 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.319973 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2137be2e64fff3c25f065874723b0549aefc64c0f9bac716ae3fd130dfeb14"} err="failed to get container status \"9f2137be2e64fff3c25f065874723b0549aefc64c0f9bac716ae3fd130dfeb14\": rpc error: code = NotFound desc = could not find container \"9f2137be2e64fff3c25f065874723b0549aefc64c0f9bac716ae3fd130dfeb14\": container with ID starting with 9f2137be2e64fff3c25f065874723b0549aefc64c0f9bac716ae3fd130dfeb14 not found: ID does not exist" Sep 19 10:48:14.320017 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.319989 2579 scope.go:117] "RemoveContainer" containerID="42dfedcc6869ef18cf27d63234a9280efb3b1f4f674dc2a0e6d48c164250059b" Sep 19 10:48:14.320381 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:48:14.320363 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42dfedcc6869ef18cf27d63234a9280efb3b1f4f674dc2a0e6d48c164250059b\": container with ID starting with 42dfedcc6869ef18cf27d63234a9280efb3b1f4f674dc2a0e6d48c164250059b not found: ID does not exist" containerID="42dfedcc6869ef18cf27d63234a9280efb3b1f4f674dc2a0e6d48c164250059b" Sep 19 10:48:14.320449 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.320386 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dfedcc6869ef18cf27d63234a9280efb3b1f4f674dc2a0e6d48c164250059b"} err="failed to get container status \"42dfedcc6869ef18cf27d63234a9280efb3b1f4f674dc2a0e6d48c164250059b\": rpc error: code = NotFound desc = could not find container \"42dfedcc6869ef18cf27d63234a9280efb3b1f4f674dc2a0e6d48c164250059b\": container with ID starting with 42dfedcc6869ef18cf27d63234a9280efb3b1f4f674dc2a0e6d48c164250059b not found: ID does not exist" Sep 19 10:48:14.386511 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.386485 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkmnt\" (UniqueName: \"kubernetes.io/projected/5074cb52-5e88-4450-89d4-bef1f809a1e3-kube-api-access-fkmnt\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:14.386511 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.386513 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5074cb52-5e88-4450-89d4-bef1f809a1e3-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:14.386719 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.386527 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:14.386719 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.386538 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:14.386719 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.386549 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5074cb52-5e88-4450-89d4-bef1f809a1e3-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:14.609319 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.609291 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr"] Sep 19 10:48:14.613870 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.613845 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr"] Sep 19 10:48:14.620303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.620276 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:48:14.620486 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.620464 2579 patch_prober.go:28] interesting pod/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 19 10:48:14.620534 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:14.620517 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" podUID="ddd05d1e-9759-421e-826f-44441269f943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 19 10:48:15.620673 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:15.620643 2579 patch_prober.go:28] interesting pod/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 19 10:48:15.621051 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:15.620694 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" podUID="ddd05d1e-9759-421e-826f-44441269f943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 19 10:48:16.196478 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:16.196446 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5074cb52-5e88-4450-89d4-bef1f809a1e3" path="/var/lib/kubelet/pods/5074cb52-5e88-4450-89d4-bef1f809a1e3/volumes" Sep 19 10:48:16.620713 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:16.620631 2579 patch_prober.go:28] interesting pod/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 19 10:48:16.620713 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:16.620692 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" podUID="ddd05d1e-9759-421e-826f-44441269f943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 19 10:48:17.620710 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:17.620669 2579 patch_prober.go:28] interesting pod/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 19 10:48:17.621157 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:17.620735 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" podUID="ddd05d1e-9759-421e-826f-44441269f943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 19 10:48:18.620910 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:18.620878 2579 patch_prober.go:28] interesting pod/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 19 10:48:18.621286 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:18.620933 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" podUID="ddd05d1e-9759-421e-826f-44441269f943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 19 10:48:19.628829 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:19.628799 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:48:19.629243 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:19.629181 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:48:19.629913 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:19.629891 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:48:25.296226 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:25.296192 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" Sep 19 10:48:33.406829 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:33.406793 2579 generic.go:356] "Generic (PLEG): container finished" podID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerID="f68e7f7003bcb3548da3a9aa06e760dd8916d82ac8380bfb046856ee2830d95c" exitCode=0 Sep 19 10:48:33.407273 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:33.406866 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" event={"ID":"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0","Type":"ContainerDied","Data":"f68e7f7003bcb3548da3a9aa06e760dd8916d82ac8380bfb046856ee2830d95c"} Sep 19 10:48:33.407335 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:33.407283 2579 scope.go:117] "RemoveContainer" containerID="f68e7f7003bcb3548da3a9aa06e760dd8916d82ac8380bfb046856ee2830d95c" Sep 19 10:48:33.408375 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:33.408348 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:33.408434 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:33.408415 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:33.408471 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:33.408443 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:34.414699 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:34.414666 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" event={"ID":"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0","Type":"ContainerStarted","Data":"9246bf21c0729738f61c5e63d4014add92775f6b6b6fb004fad1fb71e4ba185a"} Sep 19 10:48:34.858727 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:34.858626 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:34.859089 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:34.859041 2579 patch_prober.go:28] interesting pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 19 10:48:34.859402 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:34.859352 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 19 10:48:35.858363 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:35.858327 2579 patch_prober.go:28] interesting pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 19 10:48:35.858947 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:35.858914 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 19 10:48:36.858541 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:36.858507 2579 patch_prober.go:28] interesting pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 19 10:48:36.859139 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:36.859107 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 19 10:48:37.858554 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:37.858517 2579 patch_prober.go:28] interesting pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 19 10:48:37.859047 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:37.858593 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 19 10:48:38.858652 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:38.858616 2579 patch_prober.go:28] interesting pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 19 10:48:38.860028 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:38.859995 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 19 10:48:39.862477 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:39.862447 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:39.862926 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:39.862705 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:39.863531 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:39.863513 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:41.769828 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:41.769794 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb"] Sep 19 10:48:41.770457 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:41.770436 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerName="istio-proxy" Sep 19 10:48:41.770551 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:41.770459 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerName="istio-proxy" Sep 19 10:48:41.770616 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:41.770590 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerName="istio-proxy" Sep 19 10:48:41.772685 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:41.772665 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb" Sep 19 10:48:41.774956 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:41.774936 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 10:48:41.775042 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:41.774941 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 10:48:41.779278 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:41.779253 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb"] Sep 19 10:48:41.874479 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:41.874442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8623894f-6a37-4080-9007-896824c9f8ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb\" (UID: \"8623894f-6a37-4080-9007-896824c9f8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb" Sep 19 10:48:41.975553 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:41.975512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8623894f-6a37-4080-9007-896824c9f8ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb\" (UID: \"8623894f-6a37-4080-9007-896824c9f8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb" Sep 19 10:48:41.976171 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:41.976146 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8623894f-6a37-4080-9007-896824c9f8ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb\" (UID: \"8623894f-6a37-4080-9007-896824c9f8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb" Sep 19 10:48:42.084520 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:42.084432 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb" Sep 19 10:48:42.352124 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:42.352098 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb"] Sep 19 10:48:42.353989 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:48:42.353961 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8623894f_6a37_4080_9007_896824c9f8ea.slice/crio-02ef61425ff2765b293c4a6042ee8710c4f5e8845173bf2bdfb01472ab5762ae WatchSource:0}: Error finding container 02ef61425ff2765b293c4a6042ee8710c4f5e8845173bf2bdfb01472ab5762ae: Status 404 returned error can't find the container with id 02ef61425ff2765b293c4a6042ee8710c4f5e8845173bf2bdfb01472ab5762ae Sep 19 10:48:42.461174 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:42.461146 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb" event={"ID":"8623894f-6a37-4080-9007-896824c9f8ea","Type":"ContainerStarted","Data":"02ef61425ff2765b293c4a6042ee8710c4f5e8845173bf2bdfb01472ab5762ae"} Sep 19 10:48:43.467594 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:43.467565 2579 generic.go:356] "Generic (PLEG): container finished" podID="8623894f-6a37-4080-9007-896824c9f8ea" containerID="a4d7bc7b772a8008a7262ea34afadc7c8963d16156b5dc3a33107a6603b721a2" exitCode=0 Sep 19 10:48:43.467984 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:43.467651 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb" event={"ID":"8623894f-6a37-4080-9007-896824c9f8ea","Type":"ContainerDied","Data":"a4d7bc7b772a8008a7262ea34afadc7c8963d16156b5dc3a33107a6603b721a2"} Sep 19 10:48:44.601730 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:44.601701 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb" Sep 19 10:48:44.701035 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:44.701007 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8623894f-6a37-4080-9007-896824c9f8ea-maas-api-service-ca\") pod \"8623894f-6a37-4080-9007-896824c9f8ea\" (UID: \"8623894f-6a37-4080-9007-896824c9f8ea\") " Sep 19 10:48:44.701344 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:44.701323 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8623894f-6a37-4080-9007-896824c9f8ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8623894f-6a37-4080-9007-896824c9f8ea" (UID: "8623894f-6a37-4080-9007-896824c9f8ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:48:44.802405 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:44.802332 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8623894f-6a37-4080-9007-896824c9f8ea-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:45.479121 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:45.479088 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb" Sep 19 10:48:45.479121 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:45.479107 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb" event={"ID":"8623894f-6a37-4080-9007-896824c9f8ea","Type":"ContainerDied","Data":"02ef61425ff2765b293c4a6042ee8710c4f5e8845173bf2bdfb01472ab5762ae"} Sep 19 10:48:45.479340 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:45.479163 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02ef61425ff2765b293c4a6042ee8710c4f5e8845173bf2bdfb01472ab5762ae" Sep 19 10:48:53.262872 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.262832 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2"] Sep 19 10:48:53.263400 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.263113 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" podUID="ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4" containerName="payload-processing" containerID="cri-o://5ada5da35c1c6b3314a605d7856b0eaca1c3f7d932d8fce8e007476d2802dc6e" gracePeriod=30 Sep 19 10:48:53.417337 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.417263 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb"] Sep 19 10:48:53.421351 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.421326 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fb988ee6-vr8nb"] Sep 19 10:48:53.522959 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.522892 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" Sep 19 10:48:53.536240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.536208 2579 generic.go:356] "Generic (PLEG): container finished" podID="ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4" containerID="5ada5da35c1c6b3314a605d7856b0eaca1c3f7d932d8fce8e007476d2802dc6e" exitCode=0 Sep 19 10:48:53.536401 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.536249 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" Sep 19 10:48:53.536401 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.536295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" event={"ID":"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4","Type":"ContainerDied","Data":"5ada5da35c1c6b3314a605d7856b0eaca1c3f7d932d8fce8e007476d2802dc6e"} Sep 19 10:48:53.536401 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.536365 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2" event={"ID":"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4","Type":"ContainerDied","Data":"8aff1583bdc021c19ca37f671e12e479f0dc0a4d76146b8db308725c08688701"} Sep 19 10:48:53.536401 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.536393 2579 scope.go:117] "RemoveContainer" containerID="5ada5da35c1c6b3314a605d7856b0eaca1c3f7d932d8fce8e007476d2802dc6e" Sep 19 10:48:53.537428 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.537403 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw"] Sep 19 10:48:53.537679 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.537622 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerName="istio-proxy" containerID="cri-o://9246bf21c0729738f61c5e63d4014add92775f6b6b6fb004fad1fb71e4ba185a" gracePeriod=30 Sep 19 10:48:53.548893 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.548848 2579 scope.go:117] "RemoveContainer" containerID="5ada5da35c1c6b3314a605d7856b0eaca1c3f7d932d8fce8e007476d2802dc6e" Sep 19 10:48:53.549315 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:48:53.549288 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ada5da35c1c6b3314a605d7856b0eaca1c3f7d932d8fce8e007476d2802dc6e\": container with ID starting with 5ada5da35c1c6b3314a605d7856b0eaca1c3f7d932d8fce8e007476d2802dc6e not found: ID does not exist" containerID="5ada5da35c1c6b3314a605d7856b0eaca1c3f7d932d8fce8e007476d2802dc6e" Sep 19 10:48:53.549480 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.549454 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ada5da35c1c6b3314a605d7856b0eaca1c3f7d932d8fce8e007476d2802dc6e"} err="failed to get container status \"5ada5da35c1c6b3314a605d7856b0eaca1c3f7d932d8fce8e007476d2802dc6e\": rpc error: code = NotFound desc = could not find container \"5ada5da35c1c6b3314a605d7856b0eaca1c3f7d932d8fce8e007476d2802dc6e\": container with ID starting with 5ada5da35c1c6b3314a605d7856b0eaca1c3f7d932d8fce8e007476d2802dc6e not found: ID does not exist" Sep 19 10:48:53.688114 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.688082 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4-plugins-config-volume\") pod \"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4\" (UID: \"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4\") " Sep 19 10:48:53.688294 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.688134 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9zg7\" (UniqueName: \"kubernetes.io/projected/ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4-kube-api-access-g9zg7\") pod \"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4\" (UID: \"ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4\") " Sep 19 10:48:53.688458 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.688429 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4" (UID: "ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:48:53.690162 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.690141 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4-kube-api-access-g9zg7" (OuterVolumeSpecName: "kube-api-access-g9zg7") pod "ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4" (UID: "ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4"). InnerVolumeSpecName "kube-api-access-g9zg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:48:53.789085 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.788992 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:53.789085 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.789018 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9zg7\" (UniqueName: \"kubernetes.io/projected/ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4-kube-api-access-g9zg7\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:53.857549 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.857516 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2"] Sep 19 10:48:53.863378 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:53.863352 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2"] Sep 19 10:48:54.197784 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.197748 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8623894f-6a37-4080-9007-896824c9f8ea" path="/var/lib/kubelet/pods/8623894f-6a37-4080-9007-896824c9f8ea/volumes" Sep 19 10:48:54.198191 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.198176 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4" path="/var/lib/kubelet/pods/ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4/volumes" Sep 19 10:48:54.262651 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.262613 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9"] Sep 19 10:48:54.263046 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.263034 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerName="istio-proxy" Sep 19 10:48:54.263325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.263049 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerName="istio-proxy" Sep 19 10:48:54.263325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.263084 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4" containerName="payload-processing" Sep 19 10:48:54.263325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.263090 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4" containerName="payload-processing" Sep 19 10:48:54.263325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.263096 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8623894f-6a37-4080-9007-896824c9f8ea" containerName="revoke-keys" Sep 19 10:48:54.263325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.263102 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8623894f-6a37-4080-9007-896824c9f8ea" containerName="revoke-keys" Sep 19 10:48:54.263325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.263165 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5074cb52-5e88-4450-89d4-bef1f809a1e3" containerName="istio-proxy" Sep 19 10:48:54.263325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.263171 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8623894f-6a37-4080-9007-896824c9f8ea" containerName="revoke-keys" Sep 19 10:48:54.263325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.263177 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebcc856d-e6e9-41a0-8972-c8dbcf2fa4b4" containerName="payload-processing" Sep 19 10:48:54.267976 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.267958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.270561 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.270535 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-963aea34-openshift-default-dockercfg-drsfr\"" Sep 19 10:48:54.276524 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.276322 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9"] Sep 19 10:48:54.393126 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.393089 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-istio-envoy\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.393320 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.393205 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620b1788-497c-492f-996e-936f7926de3a-istio-podinfo\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.393320 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.393274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk85j\" (UniqueName: \"kubernetes.io/projected/620b1788-497c-492f-996e-936f7926de3a-kube-api-access-nk85j\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.393320 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.393311 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-workload-socket\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.393492 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.393329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620b1788-497c-492f-996e-936f7926de3a-istio-token\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.393492 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.393360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-istio-data\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.393492 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.393386 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620b1788-497c-492f-996e-936f7926de3a-istiod-ca-cert\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.393492 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.393471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-workload-certs\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.393686 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.393503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-credential-socket\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.494832 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.494726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-istio-envoy\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.494832 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.494816 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620b1788-497c-492f-996e-936f7926de3a-istio-podinfo\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.495016 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.494859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk85j\" (UniqueName: \"kubernetes.io/projected/620b1788-497c-492f-996e-936f7926de3a-kube-api-access-nk85j\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.495090 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.495006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-workload-socket\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.495090 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.495052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620b1788-497c-492f-996e-936f7926de3a-istio-token\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.495180 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.495131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-istio-data\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.495285 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.495263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620b1788-497c-492f-996e-936f7926de3a-istiod-ca-cert\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.495372 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.495319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-workload-certs\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.495372 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.495353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-credential-socket\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.495470 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.495365 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-workload-socket\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.495470 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.495410 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-istio-data\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.495653 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.495635 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-credential-socket\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.495733 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.495706 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-workload-certs\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.496009 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.495982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620b1788-497c-492f-996e-936f7926de3a-istiod-ca-cert\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.497189 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.497169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-istio-envoy\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.497420 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.497401 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620b1788-497c-492f-996e-936f7926de3a-istio-podinfo\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.502415 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.502389 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620b1788-497c-492f-996e-936f7926de3a-istio-token\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.502624 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.502600 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk85j\" (UniqueName: \"kubernetes.io/projected/620b1788-497c-492f-996e-936f7926de3a-kube-api-access-nk85j\") pod \"e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.580153 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.580119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:54.865735 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.865648 2579 patch_prober.go:28] interesting pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 19 10:48:54.865735 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.865713 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 10:48:54.870184 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.870160 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9"] Sep 19 10:48:54.871843 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:48:54.871815 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620b1788_497c_492f_996e_936f7926de3a.slice/crio-bccdd652270de50f22defbba52dbe8568cda6a72a2203bfaffcbe78141d3e999 WatchSource:0}: Error finding container bccdd652270de50f22defbba52dbe8568cda6a72a2203bfaffcbe78141d3e999: Status 404 returned error can't find the container with id bccdd652270de50f22defbba52dbe8568cda6a72a2203bfaffcbe78141d3e999 Sep 19 10:48:54.874268 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.874240 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:54.874350 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.874301 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:54.874350 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:54.874329 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:48:55.552307 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:55.552267 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" event={"ID":"620b1788-497c-492f-996e-936f7926de3a","Type":"ContainerStarted","Data":"22c357bb1a68d47e18a716c1387fe85a56837fb2ed9705d7ab3ec0e81561fa44"} Sep 19 10:48:55.552790 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:55.552349 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" event={"ID":"620b1788-497c-492f-996e-936f7926de3a","Type":"ContainerStarted","Data":"bccdd652270de50f22defbba52dbe8568cda6a72a2203bfaffcbe78141d3e999"} Sep 19 10:48:55.570789 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:55.570743 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" podStartSLOduration=1.570727122 podStartE2EDuration="1.570727122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:48:55.569002406 +0000 UTC m=+867.481507115" watchObservedRunningTime="2026-09-19 10:48:55.570727122 +0000 UTC m=+867.483231838" Sep 19 10:48:55.580329 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:55.580302 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:55.586382 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:55.586357 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:56.557810 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:56.557778 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:56.558799 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:56.558780 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:48:58.797764 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.797742 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:58.939214 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939180 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-workload-certs\") pod \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " Sep 19 10:48:58.939214 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939226 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-credential-socket\") pod \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " Sep 19 10:48:58.939449 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939246 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istiod-ca-cert\") pod \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " Sep 19 10:48:58.939449 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939269 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-envoy\") pod \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " Sep 19 10:48:58.939449 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939321 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-podinfo\") pod \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " Sep 19 10:48:58.939449 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939396 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8dkw\" (UniqueName: \"kubernetes.io/projected/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-kube-api-access-m8dkw\") pod \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " Sep 19 10:48:58.939449 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939437 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-token\") pod \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " Sep 19 10:48:58.939722 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939445 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" (UID: "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:48:58.939722 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939487 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" (UID: "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:48:58.939722 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939494 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-data\") pod \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " Sep 19 10:48:58.939722 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939617 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" (UID: "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:48:58.939722 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939664 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-workload-socket\") pod \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\" (UID: \"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0\") " Sep 19 10:48:58.939992 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939715 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" (UID: "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:48:58.939992 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.939929 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" (UID: "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:48:58.940123 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.940086 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:58.940123 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.940105 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:58.940123 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.940119 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:58.940291 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.940133 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:58.940291 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.940146 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:58.941734 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.941707 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-kube-api-access-m8dkw" (OuterVolumeSpecName: "kube-api-access-m8dkw") pod "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" (UID: "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0"). InnerVolumeSpecName "kube-api-access-m8dkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:48:58.941969 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.941945 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" (UID: "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:48:58.942041 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.941984 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" (UID: "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 10:48:58.942041 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:58.941984 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" (UID: "d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:48:59.040730 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.040691 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:59.040730 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.040723 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:59.040730 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.040736 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8dkw\" (UniqueName: \"kubernetes.io/projected/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-kube-api-access-m8dkw\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:59.040957 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.040748 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:48:59.576288 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.576256 2579 generic.go:356] "Generic (PLEG): container finished" podID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerID="9246bf21c0729738f61c5e63d4014add92775f6b6b6fb004fad1fb71e4ba185a" exitCode=0 Sep 19 10:48:59.576474 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.576307 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" Sep 19 10:48:59.576474 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.576308 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" event={"ID":"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0","Type":"ContainerDied","Data":"9246bf21c0729738f61c5e63d4014add92775f6b6b6fb004fad1fb71e4ba185a"} Sep 19 10:48:59.576474 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.576440 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw" event={"ID":"d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0","Type":"ContainerDied","Data":"9b34cc0528a49b6421fa2a9dc26beaa27c7dd5c5953746123438b533cbb36842"} Sep 19 10:48:59.576651 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.576476 2579 scope.go:117] "RemoveContainer" containerID="9246bf21c0729738f61c5e63d4014add92775f6b6b6fb004fad1fb71e4ba185a" Sep 19 10:48:59.586259 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.586245 2579 scope.go:117] "RemoveContainer" containerID="f68e7f7003bcb3548da3a9aa06e760dd8916d82ac8380bfb046856ee2830d95c" Sep 19 10:48:59.594069 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.594045 2579 scope.go:117] "RemoveContainer" containerID="9246bf21c0729738f61c5e63d4014add92775f6b6b6fb004fad1fb71e4ba185a" Sep 19 10:48:59.594340 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:48:59.594323 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9246bf21c0729738f61c5e63d4014add92775f6b6b6fb004fad1fb71e4ba185a\": container with ID starting with 9246bf21c0729738f61c5e63d4014add92775f6b6b6fb004fad1fb71e4ba185a not found: ID does not exist" containerID="9246bf21c0729738f61c5e63d4014add92775f6b6b6fb004fad1fb71e4ba185a" Sep 19 10:48:59.594393 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.594350 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9246bf21c0729738f61c5e63d4014add92775f6b6b6fb004fad1fb71e4ba185a"} err="failed to get container status \"9246bf21c0729738f61c5e63d4014add92775f6b6b6fb004fad1fb71e4ba185a\": rpc error: code = NotFound desc = could not find container \"9246bf21c0729738f61c5e63d4014add92775f6b6b6fb004fad1fb71e4ba185a\": container with ID starting with 9246bf21c0729738f61c5e63d4014add92775f6b6b6fb004fad1fb71e4ba185a not found: ID does not exist" Sep 19 10:48:59.594393 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.594368 2579 scope.go:117] "RemoveContainer" containerID="f68e7f7003bcb3548da3a9aa06e760dd8916d82ac8380bfb046856ee2830d95c" Sep 19 10:48:59.601131 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:48:59.594621 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68e7f7003bcb3548da3a9aa06e760dd8916d82ac8380bfb046856ee2830d95c\": container with ID starting with f68e7f7003bcb3548da3a9aa06e760dd8916d82ac8380bfb046856ee2830d95c not found: ID does not exist" containerID="f68e7f7003bcb3548da3a9aa06e760dd8916d82ac8380bfb046856ee2830d95c" Sep 19 10:48:59.601131 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.595912 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68e7f7003bcb3548da3a9aa06e760dd8916d82ac8380bfb046856ee2830d95c"} err="failed to get container status \"f68e7f7003bcb3548da3a9aa06e760dd8916d82ac8380bfb046856ee2830d95c\": rpc error: code = NotFound desc = could not find container \"f68e7f7003bcb3548da3a9aa06e760dd8916d82ac8380bfb046856ee2830d95c\": container with ID starting with f68e7f7003bcb3548da3a9aa06e760dd8916d82ac8380bfb046856ee2830d95c not found: ID does not exist" Sep 19 10:48:59.602949 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.602806 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw"] Sep 19 10:48:59.605754 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:48:59.605731 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw"] Sep 19 10:49:00.195538 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.195505 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" path="/var/lib/kubelet/pods/d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0/volumes" Sep 19 10:49:00.714644 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.714609 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw"] Sep 19 10:49:00.715172 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.715153 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerName="istio-proxy" Sep 19 10:49:00.715172 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.715172 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerName="istio-proxy" Sep 19 10:49:00.715326 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.715201 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerName="istio-proxy" Sep 19 10:49:00.715326 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.715210 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerName="istio-proxy" Sep 19 10:49:00.715326 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.715318 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerName="istio-proxy" Sep 19 10:49:00.719995 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.719975 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" Sep 19 10:49:00.722456 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.722434 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-963aea34-dockercfg-6kc9s\"" Sep 19 10:49:00.722559 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.722542 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-963aea34\"" Sep 19 10:49:00.727314 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.727291 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw"] Sep 19 10:49:00.749552 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.749521 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj"] Sep 19 10:49:00.750051 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.750030 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a52fa2-1f48-4ca4-a6dc-0991744dc7b0" containerName="istio-proxy" Sep 19 10:49:00.752973 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.752959 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" Sep 19 10:49:00.758342 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.758316 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2qs8\" (UniqueName: \"kubernetes.io/projected/5a19d1e9-9dcc-460e-b948-ddba56f00a60-kube-api-access-c2qs8\") pod \"payload-processing-e2e-ait-963aea34-5df966b875-fxngj\" (UID: \"5a19d1e9-9dcc-460e-b948-ddba56f00a60\") " pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" Sep 19 10:49:00.758469 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.758362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f8be43-f0bf-48ae-b3f0-0007488299d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw\" (UID: \"50f8be43-f0bf-48ae-b3f0-0007488299d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" Sep 19 10:49:00.758539 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.758482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hbnp\" (UniqueName: \"kubernetes.io/projected/50f8be43-f0bf-48ae-b3f0-0007488299d7-kube-api-access-7hbnp\") pod \"payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw\" (UID: \"50f8be43-f0bf-48ae-b3f0-0007488299d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" Sep 19 10:49:00.758539 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.758510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a19d1e9-9dcc-460e-b948-ddba56f00a60-plugins-config-volume\") pod \"payload-processing-e2e-ait-963aea34-5df966b875-fxngj\" (UID: \"5a19d1e9-9dcc-460e-b948-ddba56f00a60\") " pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" Sep 19 10:49:00.762408 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.762383 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj"] Sep 19 10:49:00.859088 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.859036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hbnp\" (UniqueName: \"kubernetes.io/projected/50f8be43-f0bf-48ae-b3f0-0007488299d7-kube-api-access-7hbnp\") pod \"payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw\" (UID: \"50f8be43-f0bf-48ae-b3f0-0007488299d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" Sep 19 10:49:00.859254 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.859101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a19d1e9-9dcc-460e-b948-ddba56f00a60-plugins-config-volume\") pod \"payload-processing-e2e-ait-963aea34-5df966b875-fxngj\" (UID: \"5a19d1e9-9dcc-460e-b948-ddba56f00a60\") " pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" Sep 19 10:49:00.859254 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.859195 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2qs8\" (UniqueName: \"kubernetes.io/projected/5a19d1e9-9dcc-460e-b948-ddba56f00a60-kube-api-access-c2qs8\") pod \"payload-processing-e2e-ait-963aea34-5df966b875-fxngj\" (UID: \"5a19d1e9-9dcc-460e-b948-ddba56f00a60\") " pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" Sep 19 10:49:00.859254 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.859230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f8be43-f0bf-48ae-b3f0-0007488299d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw\" (UID: \"50f8be43-f0bf-48ae-b3f0-0007488299d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" Sep 19 10:49:00.859978 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.859953 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a19d1e9-9dcc-460e-b948-ddba56f00a60-plugins-config-volume\") pod \"payload-processing-e2e-ait-963aea34-5df966b875-fxngj\" (UID: \"5a19d1e9-9dcc-460e-b948-ddba56f00a60\") " pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" Sep 19 10:49:00.860164 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.860143 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f8be43-f0bf-48ae-b3f0-0007488299d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw\" (UID: \"50f8be43-f0bf-48ae-b3f0-0007488299d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" Sep 19 10:49:00.866700 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.866678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hbnp\" (UniqueName: \"kubernetes.io/projected/50f8be43-f0bf-48ae-b3f0-0007488299d7-kube-api-access-7hbnp\") pod \"payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw\" (UID: \"50f8be43-f0bf-48ae-b3f0-0007488299d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" Sep 19 10:49:00.866811 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:00.866789 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2qs8\" (UniqueName: \"kubernetes.io/projected/5a19d1e9-9dcc-460e-b948-ddba56f00a60-kube-api-access-c2qs8\") pod \"payload-processing-e2e-ait-963aea34-5df966b875-fxngj\" (UID: \"5a19d1e9-9dcc-460e-b948-ddba56f00a60\") " pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" Sep 19 10:49:01.031167 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:01.031054 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" Sep 19 10:49:01.064168 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:01.064133 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" Sep 19 10:49:01.315655 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:01.315624 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw"] Sep 19 10:49:01.317801 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:49:01.317771 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f8be43_f0bf_48ae_b3f0_0007488299d7.slice/crio-ada5b7dcc23c6b160de25bbeb2371f0c04b669181ac8e74b44c2728cf53aba5c WatchSource:0}: Error finding container ada5b7dcc23c6b160de25bbeb2371f0c04b669181ac8e74b44c2728cf53aba5c: Status 404 returned error can't find the container with id ada5b7dcc23c6b160de25bbeb2371f0c04b669181ac8e74b44c2728cf53aba5c Sep 19 10:49:01.348786 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:01.348762 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj"] Sep 19 10:49:01.350624 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:49:01.350600 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a19d1e9_9dcc_460e_b948_ddba56f00a60.slice/crio-3255e58ef4804ba0f0477e011c335740529edb20bdbeb9e62b451da74d44d692 WatchSource:0}: Error finding container 3255e58ef4804ba0f0477e011c335740529edb20bdbeb9e62b451da74d44d692: Status 404 returned error can't find the container with id 3255e58ef4804ba0f0477e011c335740529edb20bdbeb9e62b451da74d44d692 Sep 19 10:49:01.591638 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:01.591548 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" event={"ID":"5a19d1e9-9dcc-460e-b948-ddba56f00a60","Type":"ContainerStarted","Data":"0c4c7fe01c1dbc96a022d1b1fdc643f79205d873d876cad2c619274c4205d496"} Sep 19 10:49:01.591638 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:01.591613 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" event={"ID":"5a19d1e9-9dcc-460e-b948-ddba56f00a60","Type":"ContainerStarted","Data":"3255e58ef4804ba0f0477e011c335740529edb20bdbeb9e62b451da74d44d692"} Sep 19 10:49:01.591856 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:01.591647 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" Sep 19 10:49:01.593193 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:01.593165 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" event={"ID":"50f8be43-f0bf-48ae-b3f0-0007488299d7","Type":"ContainerStarted","Data":"98ec42c6c43146986dd12e6b39479f13ab25723b93d45f7994eb11d88fc1d695"} Sep 19 10:49:01.593331 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:01.593206 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" event={"ID":"50f8be43-f0bf-48ae-b3f0-0007488299d7","Type":"ContainerStarted","Data":"ada5b7dcc23c6b160de25bbeb2371f0c04b669181ac8e74b44c2728cf53aba5c"} Sep 19 10:49:01.593331 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:01.593274 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" Sep 19 10:49:01.607962 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:01.607921 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" podStartSLOduration=1.6079085869999998 podStartE2EDuration="1.607908587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:49:01.607249031 +0000 UTC m=+873.519753747" watchObservedRunningTime="2026-09-19 10:49:01.607908587 +0000 UTC m=+873.520413318" Sep 19 10:49:01.624185 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:01.624146 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" podStartSLOduration=1.6241307699999998 podStartE2EDuration="1.62413077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:49:01.623559664 +0000 UTC m=+873.536064381" watchObservedRunningTime="2026-09-19 10:49:01.62413077 +0000 UTC m=+873.536635486" Sep 19 10:49:12.600238 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:12.600205 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" Sep 19 10:49:12.600688 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:12.600272 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" Sep 19 10:49:17.702005 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:17.701977 2579 generic.go:356] "Generic (PLEG): container finished" podID="620b1788-497c-492f-996e-936f7926de3a" containerID="22c357bb1a68d47e18a716c1387fe85a56837fb2ed9705d7ab3ec0e81561fa44" exitCode=0 Sep 19 10:49:17.702383 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:17.702054 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" event={"ID":"620b1788-497c-492f-996e-936f7926de3a","Type":"ContainerDied","Data":"22c357bb1a68d47e18a716c1387fe85a56837fb2ed9705d7ab3ec0e81561fa44"} Sep 19 10:49:17.702472 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:17.702458 2579 scope.go:117] "RemoveContainer" containerID="22c357bb1a68d47e18a716c1387fe85a56837fb2ed9705d7ab3ec0e81561fa44" Sep 19 10:49:17.703665 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:17.703638 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:49:17.703721 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:17.703706 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:49:17.703756 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:17.703740 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:49:18.709107 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:18.709045 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" event={"ID":"620b1788-497c-492f-996e-936f7926de3a","Type":"ContainerStarted","Data":"a61764e33d82502df7c8cb0ee6b879a33016882839c1df77dd8282f9109a71b8"} Sep 19 10:49:19.580301 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:19.580261 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:49:19.580523 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:19.580495 2579 patch_prober.go:28] interesting pod/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 19 10:49:19.580608 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:19.580569 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" podUID="620b1788-497c-492f-996e-936f7926de3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 19 10:49:20.581517 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:20.581477 2579 patch_prober.go:28] interesting pod/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 19 10:49:20.581904 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:20.581535 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" podUID="620b1788-497c-492f-996e-936f7926de3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 19 10:49:21.580977 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:21.580936 2579 patch_prober.go:28] interesting pod/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 19 10:49:21.581208 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:21.580995 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" podUID="620b1788-497c-492f-996e-936f7926de3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 19 10:49:22.581195 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:22.581160 2579 patch_prober.go:28] interesting pod/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 19 10:49:22.581650 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:22.581215 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" podUID="620b1788-497c-492f-996e-936f7926de3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 19 10:49:23.581016 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:23.580979 2579 patch_prober.go:28] interesting pod/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 19 10:49:23.581228 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:23.581033 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" podUID="620b1788-497c-492f-996e-936f7926de3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 19 10:49:24.584481 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:24.584449 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:49:24.584910 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:24.584732 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:49:24.585384 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:24.585367 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:49:27.742775 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:27.742733 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc"] Sep 19 10:49:27.745843 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:27.745821 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc" Sep 19 10:49:27.748189 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:27.748167 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 10:49:27.748277 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:27.748257 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 10:49:27.753597 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:27.753570 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc"] Sep 19 10:49:27.836913 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:27.836883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc13ca89-449d-4203-a058-9c5cbe12d07a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-963aea34-dj5bc\" (UID: \"bc13ca89-449d-4203-a058-9c5cbe12d07a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc" Sep 19 10:49:27.937709 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:27.937682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc13ca89-449d-4203-a058-9c5cbe12d07a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-963aea34-dj5bc\" (UID: \"bc13ca89-449d-4203-a058-9c5cbe12d07a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc" Sep 19 10:49:27.938304 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:27.938284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc13ca89-449d-4203-a058-9c5cbe12d07a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-963aea34-dj5bc\" (UID: \"bc13ca89-449d-4203-a058-9c5cbe12d07a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc" Sep 19 10:49:28.058301 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:28.058221 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc" Sep 19 10:49:28.326642 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:28.326616 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc"] Sep 19 10:49:28.328748 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:49:28.328724 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc13ca89_449d_4203_a058_9c5cbe12d07a.slice/crio-9c8d0301b45c7ed5587ec170c9ce18df6cee7e98a8f8e3afac5a4cd9029ff4e3 WatchSource:0}: Error finding container 9c8d0301b45c7ed5587ec170c9ce18df6cee7e98a8f8e3afac5a4cd9029ff4e3: Status 404 returned error can't find the container with id 9c8d0301b45c7ed5587ec170c9ce18df6cee7e98a8f8e3afac5a4cd9029ff4e3 Sep 19 10:49:28.767670 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:28.767637 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc" event={"ID":"bc13ca89-449d-4203-a058-9c5cbe12d07a","Type":"ContainerStarted","Data":"285f0deda549e6e0f41c584f354048acd89db896a43f990b0feb42fff9e940d7"} Sep 19 10:49:28.768035 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:28.767713 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc" event={"ID":"bc13ca89-449d-4203-a058-9c5cbe12d07a","Type":"ContainerStarted","Data":"9c8d0301b45c7ed5587ec170c9ce18df6cee7e98a8f8e3afac5a4cd9029ff4e3"} Sep 19 10:49:28.781437 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:28.781394 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc" podStartSLOduration=1.781376099 podStartE2EDuration="1.781376099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:49:28.780116593 +0000 UTC m=+900.692621310" watchObservedRunningTime="2026-09-19 10:49:28.781376099 +0000 UTC m=+900.693880814" Sep 19 10:49:29.774744 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:29.774708 2579 generic.go:356] "Generic (PLEG): container finished" podID="bc13ca89-449d-4203-a058-9c5cbe12d07a" containerID="285f0deda549e6e0f41c584f354048acd89db896a43f990b0feb42fff9e940d7" exitCode=0 Sep 19 10:49:29.775179 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:29.774801 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc" event={"ID":"bc13ca89-449d-4203-a058-9c5cbe12d07a","Type":"ContainerDied","Data":"285f0deda549e6e0f41c584f354048acd89db896a43f990b0feb42fff9e940d7"} Sep 19 10:49:30.213545 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:30.213511 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 10:49:30.214384 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:30.214358 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 10:49:30.914187 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:30.914164 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc" Sep 19 10:49:30.968002 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:30.967974 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc13ca89-449d-4203-a058-9c5cbe12d07a-maas-api-service-ca\") pod \"bc13ca89-449d-4203-a058-9c5cbe12d07a\" (UID: \"bc13ca89-449d-4203-a058-9c5cbe12d07a\") " Sep 19 10:49:30.968403 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:30.968376 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc13ca89-449d-4203-a058-9c5cbe12d07a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc13ca89-449d-4203-a058-9c5cbe12d07a" (UID: "bc13ca89-449d-4203-a058-9c5cbe12d07a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:49:31.068879 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:31.068806 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc13ca89-449d-4203-a058-9c5cbe12d07a-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:31.786525 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:31.786497 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc" Sep 19 10:49:31.786713 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:31.786543 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc" event={"ID":"bc13ca89-449d-4203-a058-9c5cbe12d07a","Type":"ContainerDied","Data":"9c8d0301b45c7ed5587ec170c9ce18df6cee7e98a8f8e3afac5a4cd9029ff4e3"} Sep 19 10:49:31.786713 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:31.786599 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c8d0301b45c7ed5587ec170c9ce18df6cee7e98a8f8e3afac5a4cd9029ff4e3" Sep 19 10:49:38.589094 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.589050 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj"] Sep 19 10:49:38.589561 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.589287 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" podUID="5a19d1e9-9dcc-460e-b948-ddba56f00a60" containerName="payload-processing" containerID="cri-o://0c4c7fe01c1dbc96a022d1b1fdc643f79205d873d876cad2c619274c4205d496" gracePeriod=30 Sep 19 10:49:38.594896 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.594844 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw"] Sep 19 10:49:38.595465 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.595430 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" podUID="50f8be43-f0bf-48ae-b3f0-0007488299d7" containerName="payload-pre-processing" containerID="cri-o://98ec42c6c43146986dd12e6b39479f13ab25723b93d45f7994eb11d88fc1d695" gracePeriod=30 Sep 19 10:49:38.726122 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.726090 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc"] Sep 19 10:49:38.731403 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.731307 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-963aea34-dj5bc"] Sep 19 10:49:38.839975 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.839888 2579 generic.go:356] "Generic (PLEG): container finished" podID="50f8be43-f0bf-48ae-b3f0-0007488299d7" containerID="98ec42c6c43146986dd12e6b39479f13ab25723b93d45f7994eb11d88fc1d695" exitCode=0 Sep 19 10:49:38.851664 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.847118 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" event={"ID":"50f8be43-f0bf-48ae-b3f0-0007488299d7","Type":"ContainerDied","Data":"98ec42c6c43146986dd12e6b39479f13ab25723b93d45f7994eb11d88fc1d695"} Sep 19 10:49:38.861538 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.861507 2579 generic.go:356] "Generic (PLEG): container finished" podID="5a19d1e9-9dcc-460e-b948-ddba56f00a60" containerID="0c4c7fe01c1dbc96a022d1b1fdc643f79205d873d876cad2c619274c4205d496" exitCode=0 Sep 19 10:49:38.861786 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.861758 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" event={"ID":"5a19d1e9-9dcc-460e-b948-ddba56f00a60","Type":"ContainerDied","Data":"0c4c7fe01c1dbc96a022d1b1fdc643f79205d873d876cad2c619274c4205d496"} Sep 19 10:49:38.887900 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.887876 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" Sep 19 10:49:38.924736 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.924706 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" Sep 19 10:49:38.953190 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.953160 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a19d1e9-9dcc-460e-b948-ddba56f00a60-plugins-config-volume\") pod \"5a19d1e9-9dcc-460e-b948-ddba56f00a60\" (UID: \"5a19d1e9-9dcc-460e-b948-ddba56f00a60\") " Sep 19 10:49:38.953323 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.953214 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f8be43-f0bf-48ae-b3f0-0007488299d7-plugins-config-volume\") pod \"50f8be43-f0bf-48ae-b3f0-0007488299d7\" (UID: \"50f8be43-f0bf-48ae-b3f0-0007488299d7\") " Sep 19 10:49:38.953323 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.953261 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2qs8\" (UniqueName: \"kubernetes.io/projected/5a19d1e9-9dcc-460e-b948-ddba56f00a60-kube-api-access-c2qs8\") pod \"5a19d1e9-9dcc-460e-b948-ddba56f00a60\" (UID: \"5a19d1e9-9dcc-460e-b948-ddba56f00a60\") " Sep 19 10:49:38.953323 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.953279 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hbnp\" (UniqueName: \"kubernetes.io/projected/50f8be43-f0bf-48ae-b3f0-0007488299d7-kube-api-access-7hbnp\") pod \"50f8be43-f0bf-48ae-b3f0-0007488299d7\" (UID: \"50f8be43-f0bf-48ae-b3f0-0007488299d7\") " Sep 19 10:49:38.953571 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.953544 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a19d1e9-9dcc-460e-b948-ddba56f00a60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a19d1e9-9dcc-460e-b948-ddba56f00a60" (UID: "5a19d1e9-9dcc-460e-b948-ddba56f00a60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:49:38.953654 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.953624 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f8be43-f0bf-48ae-b3f0-0007488299d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50f8be43-f0bf-48ae-b3f0-0007488299d7" (UID: "50f8be43-f0bf-48ae-b3f0-0007488299d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:49:38.955463 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.955442 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a19d1e9-9dcc-460e-b948-ddba56f00a60-kube-api-access-c2qs8" (OuterVolumeSpecName: "kube-api-access-c2qs8") pod "5a19d1e9-9dcc-460e-b948-ddba56f00a60" (UID: "5a19d1e9-9dcc-460e-b948-ddba56f00a60"). InnerVolumeSpecName "kube-api-access-c2qs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:49:38.956578 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:38.956556 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f8be43-f0bf-48ae-b3f0-0007488299d7-kube-api-access-7hbnp" (OuterVolumeSpecName: "kube-api-access-7hbnp") pod "50f8be43-f0bf-48ae-b3f0-0007488299d7" (UID: "50f8be43-f0bf-48ae-b3f0-0007488299d7"). InnerVolumeSpecName "kube-api-access-7hbnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:49:39.054230 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.054201 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a19d1e9-9dcc-460e-b948-ddba56f00a60-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:39.054230 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.054227 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f8be43-f0bf-48ae-b3f0-0007488299d7-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:39.054405 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.054237 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2qs8\" (UniqueName: \"kubernetes.io/projected/5a19d1e9-9dcc-460e-b948-ddba56f00a60-kube-api-access-c2qs8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:39.054405 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.054245 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hbnp\" (UniqueName: \"kubernetes.io/projected/50f8be43-f0bf-48ae-b3f0-0007488299d7-kube-api-access-7hbnp\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:39.822854 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.822816 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9"] Sep 19 10:49:39.823306 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.823001 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" podUID="620b1788-497c-492f-996e-936f7926de3a" containerName="istio-proxy" containerID="cri-o://a61764e33d82502df7c8cb0ee6b879a33016882839c1df77dd8282f9109a71b8" gracePeriod=30 Sep 19 10:49:39.868405 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.868337 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" Sep 19 10:49:39.868405 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.868380 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw" event={"ID":"50f8be43-f0bf-48ae-b3f0-0007488299d7","Type":"ContainerDied","Data":"ada5b7dcc23c6b160de25bbeb2371f0c04b669181ac8e74b44c2728cf53aba5c"} Sep 19 10:49:39.868587 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.868443 2579 scope.go:117] "RemoveContainer" containerID="98ec42c6c43146986dd12e6b39479f13ab25723b93d45f7994eb11d88fc1d695" Sep 19 10:49:39.869817 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.869799 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" Sep 19 10:49:39.869925 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.869827 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj" event={"ID":"5a19d1e9-9dcc-460e-b948-ddba56f00a60","Type":"ContainerDied","Data":"3255e58ef4804ba0f0477e011c335740529edb20bdbeb9e62b451da74d44d692"} Sep 19 10:49:39.877550 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.877536 2579 scope.go:117] "RemoveContainer" containerID="0c4c7fe01c1dbc96a022d1b1fdc643f79205d873d876cad2c619274c4205d496" Sep 19 10:49:39.893626 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.893604 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw"] Sep 19 10:49:39.896855 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.896835 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw"] Sep 19 10:49:39.912562 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.910927 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj"] Sep 19 10:49:39.914369 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:39.914344 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj"] Sep 19 10:49:40.196153 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.196125 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f8be43-f0bf-48ae-b3f0-0007488299d7" path="/var/lib/kubelet/pods/50f8be43-f0bf-48ae-b3f0-0007488299d7/volumes" Sep 19 10:49:40.196507 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.196493 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a19d1e9-9dcc-460e-b948-ddba56f00a60" path="/var/lib/kubelet/pods/5a19d1e9-9dcc-460e-b948-ddba56f00a60/volumes" Sep 19 10:49:40.196792 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.196779 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc13ca89-449d-4203-a058-9c5cbe12d07a" path="/var/lib/kubelet/pods/bc13ca89-449d-4203-a058-9c5cbe12d07a/volumes" Sep 19 10:49:40.534822 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.534731 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg"] Sep 19 10:49:40.535255 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.535239 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f8be43-f0bf-48ae-b3f0-0007488299d7" containerName="payload-pre-processing" Sep 19 10:49:40.535308 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.535256 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f8be43-f0bf-48ae-b3f0-0007488299d7" containerName="payload-pre-processing" Sep 19 10:49:40.535308 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.535275 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc13ca89-449d-4203-a058-9c5cbe12d07a" containerName="revoke-keys" Sep 19 10:49:40.535308 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.535282 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc13ca89-449d-4203-a058-9c5cbe12d07a" containerName="revoke-keys" Sep 19 10:49:40.535308 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.535289 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a19d1e9-9dcc-460e-b948-ddba56f00a60" containerName="payload-processing" Sep 19 10:49:40.535308 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.535295 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a19d1e9-9dcc-460e-b948-ddba56f00a60" containerName="payload-processing" Sep 19 10:49:40.535467 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.535351 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a19d1e9-9dcc-460e-b948-ddba56f00a60" containerName="payload-processing" Sep 19 10:49:40.535467 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.535360 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f8be43-f0bf-48ae-b3f0-0007488299d7" containerName="payload-pre-processing" Sep 19 10:49:40.535467 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.535367 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc13ca89-449d-4203-a058-9c5cbe12d07a" containerName="revoke-keys" Sep 19 10:49:40.539288 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.539263 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.541807 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.541764 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-219b8581-openshift-default-dockercfg-pgl9l\"" Sep 19 10:49:40.556074 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.556039 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg"] Sep 19 10:49:40.567805 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.567773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-data\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.567916 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.567831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww8mc\" (UniqueName: \"kubernetes.io/projected/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-kube-api-access-ww8mc\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.567916 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.567862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-workload-certs\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.568020 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.567923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-envoy\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.568020 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.567990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-podinfo\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.568124 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.568096 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-workload-socket\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.568168 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.568141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istiod-ca-cert\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.568274 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.568259 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-token\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.568344 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.568298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-credential-socket\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.669519 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.669481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-credential-socket\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.669720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.669552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-data\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.669720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.669605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww8mc\" (UniqueName: \"kubernetes.io/projected/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-kube-api-access-ww8mc\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.669904 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.669723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-workload-certs\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.669904 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.669758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-envoy\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.669904 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.669778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-podinfo\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.669904 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.669813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-workload-socket\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.669904 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.669849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istiod-ca-cert\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.670217 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.669931 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-data\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.670217 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.669958 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-credential-socket\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.670217 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.669970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-token\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.670380 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.670259 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-workload-certs\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.670380 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.670298 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-workload-socket\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.670602 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.670575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istiod-ca-cert\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.672178 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.672159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-envoy\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.672315 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.672298 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-podinfo\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.677542 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.677511 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww8mc\" (UniqueName: \"kubernetes.io/projected/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-kube-api-access-ww8mc\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.677542 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.677536 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-token\") pod \"e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:40.850799 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:40.850720 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:41.125892 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:41.125868 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg"] Sep 19 10:49:41.128051 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:49:41.128023 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe0f42d_85eb_4b35_ba0f_c5e3c000a5e7.slice/crio-0d94cff97ea38de027484c0358d5661338a7235e7c91ff0d263cc7df716d4b7e WatchSource:0}: Error finding container 0d94cff97ea38de027484c0358d5661338a7235e7c91ff0d263cc7df716d4b7e: Status 404 returned error can't find the container with id 0d94cff97ea38de027484c0358d5661338a7235e7c91ff0d263cc7df716d4b7e Sep 19 10:49:41.130448 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:41.130416 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:49:41.130535 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:41.130481 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:49:41.130535 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:41.130509 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:49:41.884673 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:41.884641 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" event={"ID":"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7","Type":"ContainerStarted","Data":"88009ddc08b425e8f2632d3dfb39eb08a8309ec8d102b18ae3893566a2f6d879"} Sep 19 10:49:41.885054 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:41.884697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" event={"ID":"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7","Type":"ContainerStarted","Data":"0d94cff97ea38de027484c0358d5661338a7235e7c91ff0d263cc7df716d4b7e"} Sep 19 10:49:41.902932 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:41.902886 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" podStartSLOduration=1.902872036 podStartE2EDuration="1.902872036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:49:41.901680757 +0000 UTC m=+913.814185498" watchObservedRunningTime="2026-09-19 10:49:41.902872036 +0000 UTC m=+913.815376752" Sep 19 10:49:42.851528 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:42.851492 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:42.857104 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:42.857078 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:42.890409 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:42.890382 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:42.891197 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:42.891176 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:49:45.085026 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.084966 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:49:45.218957 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.218923 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk85j\" (UniqueName: \"kubernetes.io/projected/620b1788-497c-492f-996e-936f7926de3a-kube-api-access-nk85j\") pod \"620b1788-497c-492f-996e-936f7926de3a\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " Sep 19 10:49:45.219148 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.218998 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620b1788-497c-492f-996e-936f7926de3a-istio-token\") pod \"620b1788-497c-492f-996e-936f7926de3a\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " Sep 19 10:49:45.219148 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219048 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620b1788-497c-492f-996e-936f7926de3a-istio-podinfo\") pod \"620b1788-497c-492f-996e-936f7926de3a\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " Sep 19 10:49:45.219148 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219118 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620b1788-497c-492f-996e-936f7926de3a-istiod-ca-cert\") pod \"620b1788-497c-492f-996e-936f7926de3a\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " Sep 19 10:49:45.219305 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219203 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-workload-socket\") pod \"620b1788-497c-492f-996e-936f7926de3a\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " Sep 19 10:49:45.219305 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219226 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-istio-data\") pod \"620b1788-497c-492f-996e-936f7926de3a\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " Sep 19 10:49:45.219305 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219262 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-istio-envoy\") pod \"620b1788-497c-492f-996e-936f7926de3a\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " Sep 19 10:49:45.219305 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219291 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-workload-certs\") pod \"620b1788-497c-492f-996e-936f7926de3a\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " Sep 19 10:49:45.219504 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219314 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-credential-socket\") pod \"620b1788-497c-492f-996e-936f7926de3a\" (UID: \"620b1788-497c-492f-996e-936f7926de3a\") " Sep 19 10:49:45.219558 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219520 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "620b1788-497c-492f-996e-936f7926de3a" (UID: "620b1788-497c-492f-996e-936f7926de3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:49:45.219558 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219515 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-istio-data" (OuterVolumeSpecName: "istio-data") pod "620b1788-497c-492f-996e-936f7926de3a" (UID: "620b1788-497c-492f-996e-936f7926de3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:49:45.219660 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219631 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:45.219660 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219650 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:45.219660 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219628 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620b1788-497c-492f-996e-936f7926de3a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "620b1788-497c-492f-996e-936f7926de3a" (UID: "620b1788-497c-492f-996e-936f7926de3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:49:45.219804 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219718 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "620b1788-497c-492f-996e-936f7926de3a" (UID: "620b1788-497c-492f-996e-936f7926de3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:49:45.219804 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.219733 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "620b1788-497c-492f-996e-936f7926de3a" (UID: "620b1788-497c-492f-996e-936f7926de3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:49:45.221270 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.221246 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620b1788-497c-492f-996e-936f7926de3a-kube-api-access-nk85j" (OuterVolumeSpecName: "kube-api-access-nk85j") pod "620b1788-497c-492f-996e-936f7926de3a" (UID: "620b1788-497c-492f-996e-936f7926de3a"). InnerVolumeSpecName "kube-api-access-nk85j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:49:45.221441 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.221420 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "620b1788-497c-492f-996e-936f7926de3a" (UID: "620b1788-497c-492f-996e-936f7926de3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:49:45.221611 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.221594 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/620b1788-497c-492f-996e-936f7926de3a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "620b1788-497c-492f-996e-936f7926de3a" (UID: "620b1788-497c-492f-996e-936f7926de3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 10:49:45.221656 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.221638 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620b1788-497c-492f-996e-936f7926de3a-istio-token" (OuterVolumeSpecName: "istio-token") pod "620b1788-497c-492f-996e-936f7926de3a" (UID: "620b1788-497c-492f-996e-936f7926de3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:49:45.320645 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.320618 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:45.320645 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.320639 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:45.320806 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.320652 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620b1788-497c-492f-996e-936f7926de3a-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:45.320806 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.320666 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk85j\" (UniqueName: \"kubernetes.io/projected/620b1788-497c-492f-996e-936f7926de3a-kube-api-access-nk85j\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:45.320806 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.320675 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620b1788-497c-492f-996e-936f7926de3a-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:45.320806 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.320683 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620b1788-497c-492f-996e-936f7926de3a-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:45.320806 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.320690 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620b1788-497c-492f-996e-936f7926de3a-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:49:45.909557 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.909527 2579 generic.go:356] "Generic (PLEG): container finished" podID="620b1788-497c-492f-996e-936f7926de3a" containerID="a61764e33d82502df7c8cb0ee6b879a33016882839c1df77dd8282f9109a71b8" exitCode=0 Sep 19 10:49:45.909706 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.909577 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" Sep 19 10:49:45.909706 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.909583 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" event={"ID":"620b1788-497c-492f-996e-936f7926de3a","Type":"ContainerDied","Data":"a61764e33d82502df7c8cb0ee6b879a33016882839c1df77dd8282f9109a71b8"} Sep 19 10:49:45.909706 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.909700 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9" event={"ID":"620b1788-497c-492f-996e-936f7926de3a","Type":"ContainerDied","Data":"bccdd652270de50f22defbba52dbe8568cda6a72a2203bfaffcbe78141d3e999"} Sep 19 10:49:45.909823 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.909726 2579 scope.go:117] "RemoveContainer" containerID="a61764e33d82502df7c8cb0ee6b879a33016882839c1df77dd8282f9109a71b8" Sep 19 10:49:45.920470 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.920435 2579 scope.go:117] "RemoveContainer" containerID="22c357bb1a68d47e18a716c1387fe85a56837fb2ed9705d7ab3ec0e81561fa44" Sep 19 10:49:45.931382 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.931362 2579 scope.go:117] "RemoveContainer" containerID="a61764e33d82502df7c8cb0ee6b879a33016882839c1df77dd8282f9109a71b8" Sep 19 10:49:45.931633 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:49:45.931617 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61764e33d82502df7c8cb0ee6b879a33016882839c1df77dd8282f9109a71b8\": container with ID starting with a61764e33d82502df7c8cb0ee6b879a33016882839c1df77dd8282f9109a71b8 not found: ID does not exist" containerID="a61764e33d82502df7c8cb0ee6b879a33016882839c1df77dd8282f9109a71b8" Sep 19 10:49:45.931680 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.931640 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61764e33d82502df7c8cb0ee6b879a33016882839c1df77dd8282f9109a71b8"} err="failed to get container status \"a61764e33d82502df7c8cb0ee6b879a33016882839c1df77dd8282f9109a71b8\": rpc error: code = NotFound desc = could not find container \"a61764e33d82502df7c8cb0ee6b879a33016882839c1df77dd8282f9109a71b8\": container with ID starting with a61764e33d82502df7c8cb0ee6b879a33016882839c1df77dd8282f9109a71b8 not found: ID does not exist" Sep 19 10:49:45.931680 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.931655 2579 scope.go:117] "RemoveContainer" containerID="22c357bb1a68d47e18a716c1387fe85a56837fb2ed9705d7ab3ec0e81561fa44" Sep 19 10:49:45.931909 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:49:45.931891 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c357bb1a68d47e18a716c1387fe85a56837fb2ed9705d7ab3ec0e81561fa44\": container with ID starting with 22c357bb1a68d47e18a716c1387fe85a56837fb2ed9705d7ab3ec0e81561fa44 not found: ID does not exist" containerID="22c357bb1a68d47e18a716c1387fe85a56837fb2ed9705d7ab3ec0e81561fa44" Sep 19 10:49:45.931948 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.931915 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c357bb1a68d47e18a716c1387fe85a56837fb2ed9705d7ab3ec0e81561fa44"} err="failed to get container status \"22c357bb1a68d47e18a716c1387fe85a56837fb2ed9705d7ab3ec0e81561fa44\": rpc error: code = NotFound desc = could not find container \"22c357bb1a68d47e18a716c1387fe85a56837fb2ed9705d7ab3ec0e81561fa44\": container with ID starting with 22c357bb1a68d47e18a716c1387fe85a56837fb2ed9705d7ab3ec0e81561fa44 not found: ID does not exist" Sep 19 10:49:45.936764 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.936742 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9"] Sep 19 10:49:45.940424 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:45.940404 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9"] Sep 19 10:49:46.197496 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.197421 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="620b1788-497c-492f-996e-936f7926de3a" path="/var/lib/kubelet/pods/620b1788-497c-492f-996e-936f7926de3a/volumes" Sep 19 10:49:46.514959 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.514892 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k"] Sep 19 10:49:46.515352 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.515337 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620b1788-497c-492f-996e-936f7926de3a" containerName="istio-proxy" Sep 19 10:49:46.515402 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.515353 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="620b1788-497c-492f-996e-936f7926de3a" containerName="istio-proxy" Sep 19 10:49:46.515402 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.515370 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620b1788-497c-492f-996e-936f7926de3a" containerName="istio-proxy" Sep 19 10:49:46.515402 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.515376 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="620b1788-497c-492f-996e-936f7926de3a" containerName="istio-proxy" Sep 19 10:49:46.515505 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.515437 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="620b1788-497c-492f-996e-936f7926de3a" containerName="istio-proxy" Sep 19 10:49:46.515505 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.515445 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="620b1788-497c-492f-996e-936f7926de3a" containerName="istio-proxy" Sep 19 10:49:46.519907 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.519889 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" Sep 19 10:49:46.522381 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.522360 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-219b8581\"" Sep 19 10:49:46.522613 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.522600 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-219b8581-dockercfg-gmnw6\"" Sep 19 10:49:46.528724 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.528698 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k"] Sep 19 10:49:46.555385 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.555351 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr"] Sep 19 10:49:46.559684 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.559666 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" Sep 19 10:49:46.566220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.566175 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr"] Sep 19 10:49:46.635022 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.634985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee53b8b-464c-42b8-bea2-34e17646094f-plugins-config-volume\") pod \"payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr\" (UID: \"7ee53b8b-464c-42b8-bea2-34e17646094f\") " pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" Sep 19 10:49:46.635361 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.635339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-944lh\" (UniqueName: \"kubernetes.io/projected/7ee53b8b-464c-42b8-bea2-34e17646094f-kube-api-access-944lh\") pod \"payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr\" (UID: \"7ee53b8b-464c-42b8-bea2-34e17646094f\") " pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" Sep 19 10:49:46.635509 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.635492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns6s5\" (UniqueName: \"kubernetes.io/projected/4830ab34-a086-4653-9db3-6f63a1e7286b-kube-api-access-ns6s5\") pod \"payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k\" (UID: \"4830ab34-a086-4653-9db3-6f63a1e7286b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" Sep 19 10:49:46.635680 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.635661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830ab34-a086-4653-9db3-6f63a1e7286b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k\" (UID: \"4830ab34-a086-4653-9db3-6f63a1e7286b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" Sep 19 10:49:46.736964 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.736935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee53b8b-464c-42b8-bea2-34e17646094f-plugins-config-volume\") pod \"payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr\" (UID: \"7ee53b8b-464c-42b8-bea2-34e17646094f\") " pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" Sep 19 10:49:46.737161 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.736984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-944lh\" (UniqueName: \"kubernetes.io/projected/7ee53b8b-464c-42b8-bea2-34e17646094f-kube-api-access-944lh\") pod \"payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr\" (UID: \"7ee53b8b-464c-42b8-bea2-34e17646094f\") " pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" Sep 19 10:49:46.737161 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.737085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns6s5\" (UniqueName: \"kubernetes.io/projected/4830ab34-a086-4653-9db3-6f63a1e7286b-kube-api-access-ns6s5\") pod \"payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k\" (UID: \"4830ab34-a086-4653-9db3-6f63a1e7286b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" Sep 19 10:49:46.737161 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.737108 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830ab34-a086-4653-9db3-6f63a1e7286b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k\" (UID: \"4830ab34-a086-4653-9db3-6f63a1e7286b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" Sep 19 10:49:46.737738 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.737713 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee53b8b-464c-42b8-bea2-34e17646094f-plugins-config-volume\") pod \"payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr\" (UID: \"7ee53b8b-464c-42b8-bea2-34e17646094f\") " pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" Sep 19 10:49:46.737823 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.737757 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830ab34-a086-4653-9db3-6f63a1e7286b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k\" (UID: \"4830ab34-a086-4653-9db3-6f63a1e7286b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" Sep 19 10:49:46.744819 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.744797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-944lh\" (UniqueName: \"kubernetes.io/projected/7ee53b8b-464c-42b8-bea2-34e17646094f-kube-api-access-944lh\") pod \"payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr\" (UID: \"7ee53b8b-464c-42b8-bea2-34e17646094f\") " pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" Sep 19 10:49:46.744923 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.744804 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns6s5\" (UniqueName: \"kubernetes.io/projected/4830ab34-a086-4653-9db3-6f63a1e7286b-kube-api-access-ns6s5\") pod \"payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k\" (UID: \"4830ab34-a086-4653-9db3-6f63a1e7286b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" Sep 19 10:49:46.831755 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.831679 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" Sep 19 10:49:46.871424 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:46.871388 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" Sep 19 10:49:47.102696 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:47.102669 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k"] Sep 19 10:49:47.104220 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:49:47.104190 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4830ab34_a086_4653_9db3_6f63a1e7286b.slice/crio-b7661381c90c51e90acec7996ebf4e8e09efeb087b7eb7b803b11172d9c39aff WatchSource:0}: Error finding container b7661381c90c51e90acec7996ebf4e8e09efeb087b7eb7b803b11172d9c39aff: Status 404 returned error can't find the container with id b7661381c90c51e90acec7996ebf4e8e09efeb087b7eb7b803b11172d9c39aff Sep 19 10:49:47.158376 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:47.158353 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr"] Sep 19 10:49:47.161224 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:49:47.161202 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee53b8b_464c_42b8_bea2_34e17646094f.slice/crio-43478a3232968e6eeea5aa88b5fa3c081b62418c0453415a0fb4c53b6d4be846 WatchSource:0}: Error finding container 43478a3232968e6eeea5aa88b5fa3c081b62418c0453415a0fb4c53b6d4be846: Status 404 returned error can't find the container with id 43478a3232968e6eeea5aa88b5fa3c081b62418c0453415a0fb4c53b6d4be846 Sep 19 10:49:47.927152 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:47.927118 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" event={"ID":"4830ab34-a086-4653-9db3-6f63a1e7286b","Type":"ContainerStarted","Data":"25e5a424a7a814d8b12c96a1e952f6b0a5b1c0772d26ee5faff450da55a38e15"} Sep 19 10:49:47.927594 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:47.927186 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" Sep 19 10:49:47.927594 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:47.927213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" event={"ID":"4830ab34-a086-4653-9db3-6f63a1e7286b","Type":"ContainerStarted","Data":"b7661381c90c51e90acec7996ebf4e8e09efeb087b7eb7b803b11172d9c39aff"} Sep 19 10:49:47.928851 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:47.928822 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" event={"ID":"7ee53b8b-464c-42b8-bea2-34e17646094f","Type":"ContainerStarted","Data":"ea08e759049d90f7235e4dd5c451749fed6f479711e025ee3faa2daefd7a2321"} Sep 19 10:49:47.928987 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:47.928871 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" event={"ID":"7ee53b8b-464c-42b8-bea2-34e17646094f","Type":"ContainerStarted","Data":"43478a3232968e6eeea5aa88b5fa3c081b62418c0453415a0fb4c53b6d4be846"} Sep 19 10:49:47.928987 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:47.928901 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" Sep 19 10:49:47.943380 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:47.943327 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" podStartSLOduration=1.943307307 podStartE2EDuration="1.943307307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:49:47.941933542 +0000 UTC m=+919.854438260" watchObservedRunningTime="2026-09-19 10:49:47.943307307 +0000 UTC m=+919.855812024" Sep 19 10:49:47.960243 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:47.960192 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" podStartSLOduration=1.9601724200000001 podStartE2EDuration="1.96017242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:49:47.956633786 +0000 UTC m=+919.869138504" watchObservedRunningTime="2026-09-19 10:49:47.96017242 +0000 UTC m=+919.872677142" Sep 19 10:49:58.937088 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:58.937038 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" Sep 19 10:49:58.937537 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:49:58.937213 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" Sep 19 10:50:08.064919 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:08.064874 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl"] Sep 19 10:50:08.069172 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:08.069143 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl" Sep 19 10:50:08.071827 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:08.071800 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 10:50:08.072885 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:08.072864 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 10:50:08.073498 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:08.073472 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl"] Sep 19 10:50:08.160766 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:08.160635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d4859e-7650-4582-8f3f-81f4f4440661-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-219b8581-mnnhl\" (UID: \"27d4859e-7650-4582-8f3f-81f4f4440661\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl" Sep 19 10:50:08.263482 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:08.262413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d4859e-7650-4582-8f3f-81f4f4440661-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-219b8581-mnnhl\" (UID: \"27d4859e-7650-4582-8f3f-81f4f4440661\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl" Sep 19 10:50:08.263482 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:08.263427 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d4859e-7650-4582-8f3f-81f4f4440661-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-219b8581-mnnhl\" (UID: \"27d4859e-7650-4582-8f3f-81f4f4440661\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl" Sep 19 10:50:08.382192 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:08.382103 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl" Sep 19 10:50:08.660231 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:08.660193 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl"] Sep 19 10:50:08.669742 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:50:08.669714 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d4859e_7650_4582_8f3f_81f4f4440661.slice/crio-97143b72f379fe625d461932b0f8294a6899921fffb64d06f9298ee5c7abc189 WatchSource:0}: Error finding container 97143b72f379fe625d461932b0f8294a6899921fffb64d06f9298ee5c7abc189: Status 404 returned error can't find the container with id 97143b72f379fe625d461932b0f8294a6899921fffb64d06f9298ee5c7abc189 Sep 19 10:50:09.054616 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:09.054565 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl" event={"ID":"27d4859e-7650-4582-8f3f-81f4f4440661","Type":"ContainerStarted","Data":"97143b72f379fe625d461932b0f8294a6899921fffb64d06f9298ee5c7abc189"} Sep 19 10:50:10.064844 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:10.064747 2579 generic.go:356] "Generic (PLEG): container finished" podID="27d4859e-7650-4582-8f3f-81f4f4440661" containerID="44bd53cfd09158c76d50becf9f153380f3a5d1265a3f8e1441673145998b8a61" exitCode=0 Sep 19 10:50:10.064844 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:10.064808 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl" event={"ID":"27d4859e-7650-4582-8f3f-81f4f4440661","Type":"ContainerDied","Data":"44bd53cfd09158c76d50becf9f153380f3a5d1265a3f8e1441673145998b8a61"} Sep 19 10:50:11.231327 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:11.231303 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl" Sep 19 10:50:11.308984 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:11.308954 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d4859e-7650-4582-8f3f-81f4f4440661-maas-api-service-ca\") pod \"27d4859e-7650-4582-8f3f-81f4f4440661\" (UID: \"27d4859e-7650-4582-8f3f-81f4f4440661\") " Sep 19 10:50:11.309303 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:11.309282 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d4859e-7650-4582-8f3f-81f4f4440661-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27d4859e-7650-4582-8f3f-81f4f4440661" (UID: "27d4859e-7650-4582-8f3f-81f4f4440661"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:50:11.409874 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:11.409847 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d4859e-7650-4582-8f3f-81f4f4440661-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:12.080211 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:12.080180 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl" Sep 19 10:50:12.080393 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:12.080226 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl" event={"ID":"27d4859e-7650-4582-8f3f-81f4f4440661","Type":"ContainerDied","Data":"97143b72f379fe625d461932b0f8294a6899921fffb64d06f9298ee5c7abc189"} Sep 19 10:50:12.080393 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:12.080270 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97143b72f379fe625d461932b0f8294a6899921fffb64d06f9298ee5c7abc189" Sep 19 10:50:13.892090 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:13.892040 2579 patch_prober.go:28] interesting pod/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 19 10:50:13.892468 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:13.892120 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" podUID="dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 19 10:50:15.098647 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.098610 2579 generic.go:356] "Generic (PLEG): container finished" podID="dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" containerID="88009ddc08b425e8f2632d3dfb39eb08a8309ec8d102b18ae3893566a2f6d879" exitCode=0 Sep 19 10:50:15.099103 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.098679 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" event={"ID":"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7","Type":"ContainerDied","Data":"88009ddc08b425e8f2632d3dfb39eb08a8309ec8d102b18ae3893566a2f6d879"} Sep 19 10:50:15.099208 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.099192 2579 scope.go:117] "RemoveContainer" containerID="88009ddc08b425e8f2632d3dfb39eb08a8309ec8d102b18ae3893566a2f6d879" Sep 19 10:50:15.100350 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.100322 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:50:15.100445 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.100397 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:50:15.100445 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.100436 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:50:15.217159 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.217123 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr"] Sep 19 10:50:15.217697 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.217658 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" podUID="7ee53b8b-464c-42b8-bea2-34e17646094f" containerName="payload-processing" containerID="cri-o://ea08e759049d90f7235e4dd5c451749fed6f479711e025ee3faa2daefd7a2321" gracePeriod=30 Sep 19 10:50:15.228112 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.226483 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k"] Sep 19 10:50:15.228112 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.226696 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" podUID="4830ab34-a086-4653-9db3-6f63a1e7286b" containerName="payload-pre-processing" containerID="cri-o://25e5a424a7a814d8b12c96a1e952f6b0a5b1c0772d26ee5faff450da55a38e15" gracePeriod=30 Sep 19 10:50:15.365262 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.365229 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl"] Sep 19 10:50:15.371089 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.371043 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-219b8581-mnnhl"] Sep 19 10:50:15.501607 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.501572 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg"] Sep 19 10:50:15.570432 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.570409 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" Sep 19 10:50:15.596675 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.596409 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" Sep 19 10:50:15.666794 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.666756 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns6s5\" (UniqueName: \"kubernetes.io/projected/4830ab34-a086-4653-9db3-6f63a1e7286b-kube-api-access-ns6s5\") pod \"4830ab34-a086-4653-9db3-6f63a1e7286b\" (UID: \"4830ab34-a086-4653-9db3-6f63a1e7286b\") " Sep 19 10:50:15.666997 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.666833 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830ab34-a086-4653-9db3-6f63a1e7286b-plugins-config-volume\") pod \"4830ab34-a086-4653-9db3-6f63a1e7286b\" (UID: \"4830ab34-a086-4653-9db3-6f63a1e7286b\") " Sep 19 10:50:15.666997 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.666861 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee53b8b-464c-42b8-bea2-34e17646094f-plugins-config-volume\") pod \"7ee53b8b-464c-42b8-bea2-34e17646094f\" (UID: \"7ee53b8b-464c-42b8-bea2-34e17646094f\") " Sep 19 10:50:15.666997 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.666918 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-944lh\" (UniqueName: \"kubernetes.io/projected/7ee53b8b-464c-42b8-bea2-34e17646094f-kube-api-access-944lh\") pod \"7ee53b8b-464c-42b8-bea2-34e17646094f\" (UID: \"7ee53b8b-464c-42b8-bea2-34e17646094f\") " Sep 19 10:50:15.667328 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.667298 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4830ab34-a086-4653-9db3-6f63a1e7286b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4830ab34-a086-4653-9db3-6f63a1e7286b" (UID: "4830ab34-a086-4653-9db3-6f63a1e7286b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:50:15.667449 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.667301 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee53b8b-464c-42b8-bea2-34e17646094f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ee53b8b-464c-42b8-bea2-34e17646094f" (UID: "7ee53b8b-464c-42b8-bea2-34e17646094f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:50:15.669024 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.669004 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4830ab34-a086-4653-9db3-6f63a1e7286b-kube-api-access-ns6s5" (OuterVolumeSpecName: "kube-api-access-ns6s5") pod "4830ab34-a086-4653-9db3-6f63a1e7286b" (UID: "4830ab34-a086-4653-9db3-6f63a1e7286b"). InnerVolumeSpecName "kube-api-access-ns6s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:50:15.669293 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.669273 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee53b8b-464c-42b8-bea2-34e17646094f-kube-api-access-944lh" (OuterVolumeSpecName: "kube-api-access-944lh") pod "7ee53b8b-464c-42b8-bea2-34e17646094f" (UID: "7ee53b8b-464c-42b8-bea2-34e17646094f"). InnerVolumeSpecName "kube-api-access-944lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:50:15.768757 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.768726 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-944lh\" (UniqueName: \"kubernetes.io/projected/7ee53b8b-464c-42b8-bea2-34e17646094f-kube-api-access-944lh\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:15.768757 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.768753 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns6s5\" (UniqueName: \"kubernetes.io/projected/4830ab34-a086-4653-9db3-6f63a1e7286b-kube-api-access-ns6s5\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:15.768757 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.768763 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830ab34-a086-4653-9db3-6f63a1e7286b-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:15.769025 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:15.768772 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee53b8b-464c-42b8-bea2-34e17646094f-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:16.108673 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.108589 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" event={"ID":"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7","Type":"ContainerStarted","Data":"b7e30300aead0f42a1f0fcb27dfa07c1c7f20f1ec395ecfd8687ba38f8e0e7a8"} Sep 19 10:50:16.109126 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.108668 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" podUID="dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" containerName="istio-proxy" containerID="cri-o://b7e30300aead0f42a1f0fcb27dfa07c1c7f20f1ec395ecfd8687ba38f8e0e7a8" gracePeriod=30 Sep 19 10:50:16.110205 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.110184 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" Sep 19 10:50:16.110205 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.110189 2579 generic.go:356] "Generic (PLEG): container finished" podID="4830ab34-a086-4653-9db3-6f63a1e7286b" containerID="25e5a424a7a814d8b12c96a1e952f6b0a5b1c0772d26ee5faff450da55a38e15" exitCode=0 Sep 19 10:50:16.110787 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.110224 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" event={"ID":"4830ab34-a086-4653-9db3-6f63a1e7286b","Type":"ContainerDied","Data":"25e5a424a7a814d8b12c96a1e952f6b0a5b1c0772d26ee5faff450da55a38e15"} Sep 19 10:50:16.110787 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.110279 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k" event={"ID":"4830ab34-a086-4653-9db3-6f63a1e7286b","Type":"ContainerDied","Data":"b7661381c90c51e90acec7996ebf4e8e09efeb087b7eb7b803b11172d9c39aff"} Sep 19 10:50:16.110787 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.110479 2579 scope.go:117] "RemoveContainer" containerID="25e5a424a7a814d8b12c96a1e952f6b0a5b1c0772d26ee5faff450da55a38e15" Sep 19 10:50:16.111858 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.111837 2579 generic.go:356] "Generic (PLEG): container finished" podID="7ee53b8b-464c-42b8-bea2-34e17646094f" containerID="ea08e759049d90f7235e4dd5c451749fed6f479711e025ee3faa2daefd7a2321" exitCode=0 Sep 19 10:50:16.111958 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.111867 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" event={"ID":"7ee53b8b-464c-42b8-bea2-34e17646094f","Type":"ContainerDied","Data":"ea08e759049d90f7235e4dd5c451749fed6f479711e025ee3faa2daefd7a2321"} Sep 19 10:50:16.111958 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.111937 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" event={"ID":"7ee53b8b-464c-42b8-bea2-34e17646094f","Type":"ContainerDied","Data":"43478a3232968e6eeea5aa88b5fa3c081b62418c0453415a0fb4c53b6d4be846"} Sep 19 10:50:16.112075 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.111887 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr" Sep 19 10:50:16.120582 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.120490 2579 scope.go:117] "RemoveContainer" containerID="25e5a424a7a814d8b12c96a1e952f6b0a5b1c0772d26ee5faff450da55a38e15" Sep 19 10:50:16.120797 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:50:16.120776 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e5a424a7a814d8b12c96a1e952f6b0a5b1c0772d26ee5faff450da55a38e15\": container with ID starting with 25e5a424a7a814d8b12c96a1e952f6b0a5b1c0772d26ee5faff450da55a38e15 not found: ID does not exist" containerID="25e5a424a7a814d8b12c96a1e952f6b0a5b1c0772d26ee5faff450da55a38e15" Sep 19 10:50:16.120874 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.120804 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e5a424a7a814d8b12c96a1e952f6b0a5b1c0772d26ee5faff450da55a38e15"} err="failed to get container status \"25e5a424a7a814d8b12c96a1e952f6b0a5b1c0772d26ee5faff450da55a38e15\": rpc error: code = NotFound desc = could not find container \"25e5a424a7a814d8b12c96a1e952f6b0a5b1c0772d26ee5faff450da55a38e15\": container with ID starting with 25e5a424a7a814d8b12c96a1e952f6b0a5b1c0772d26ee5faff450da55a38e15 not found: ID does not exist" Sep 19 10:50:16.120874 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.120823 2579 scope.go:117] "RemoveContainer" containerID="ea08e759049d90f7235e4dd5c451749fed6f479711e025ee3faa2daefd7a2321" Sep 19 10:50:16.129540 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.129524 2579 scope.go:117] "RemoveContainer" containerID="ea08e759049d90f7235e4dd5c451749fed6f479711e025ee3faa2daefd7a2321" Sep 19 10:50:16.129773 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:50:16.129754 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea08e759049d90f7235e4dd5c451749fed6f479711e025ee3faa2daefd7a2321\": container with ID starting with ea08e759049d90f7235e4dd5c451749fed6f479711e025ee3faa2daefd7a2321 not found: ID does not exist" containerID="ea08e759049d90f7235e4dd5c451749fed6f479711e025ee3faa2daefd7a2321" Sep 19 10:50:16.129825 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.129781 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea08e759049d90f7235e4dd5c451749fed6f479711e025ee3faa2daefd7a2321"} err="failed to get container status \"ea08e759049d90f7235e4dd5c451749fed6f479711e025ee3faa2daefd7a2321\": rpc error: code = NotFound desc = could not find container \"ea08e759049d90f7235e4dd5c451749fed6f479711e025ee3faa2daefd7a2321\": container with ID starting with ea08e759049d90f7235e4dd5c451749fed6f479711e025ee3faa2daefd7a2321 not found: ID does not exist" Sep 19 10:50:16.157896 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.157868 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr"] Sep 19 10:50:16.163889 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.163865 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr"] Sep 19 10:50:16.174290 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.174269 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k"] Sep 19 10:50:16.178111 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.178090 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k"] Sep 19 10:50:16.196029 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.196004 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27d4859e-7650-4582-8f3f-81f4f4440661" path="/var/lib/kubelet/pods/27d4859e-7650-4582-8f3f-81f4f4440661/volumes" Sep 19 10:50:16.196421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.196407 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4830ab34-a086-4653-9db3-6f63a1e7286b" path="/var/lib/kubelet/pods/4830ab34-a086-4653-9db3-6f63a1e7286b/volumes" Sep 19 10:50:16.196756 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:16.196744 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee53b8b-464c-42b8-bea2-34e17646094f" path="/var/lib/kubelet/pods/7ee53b8b-464c-42b8-bea2-34e17646094f/volumes" Sep 19 10:50:21.368980 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.368925 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:50:21.421383 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.421351 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-token\") pod \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " Sep 19 10:50:21.421526 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.421406 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-workload-certs\") pod \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " Sep 19 10:50:21.421526 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.421450 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istiod-ca-cert\") pod \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " Sep 19 10:50:21.421526 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.421479 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww8mc\" (UniqueName: \"kubernetes.io/projected/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-kube-api-access-ww8mc\") pod \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " Sep 19 10:50:21.421666 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.421548 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-data\") pod \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " Sep 19 10:50:21.421666 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.421577 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-workload-socket\") pod \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " Sep 19 10:50:21.421666 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.421638 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-envoy\") pod \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " Sep 19 10:50:21.421828 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.421678 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-podinfo\") pod \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " Sep 19 10:50:21.421828 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.421707 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-credential-socket\") pod \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\" (UID: \"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7\") " Sep 19 10:50:21.421828 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.421746 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" (UID: "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:50:21.421979 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.421896 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" (UID: "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:50:21.421979 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.421925 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" (UID: "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:50:21.422199 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.422180 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:21.422274 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.422205 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:21.422274 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.422237 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:21.422274 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.422200 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" (UID: "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:50:21.422274 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.422242 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" (UID: "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:50:21.423768 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.423738 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-kube-api-access-ww8mc" (OuterVolumeSpecName: "kube-api-access-ww8mc") pod "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" (UID: "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7"). InnerVolumeSpecName "kube-api-access-ww8mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:50:21.424194 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.424174 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" (UID: "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:50:21.424274 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.424212 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" (UID: "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:50:21.424333 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.424315 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" (UID: "dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 10:50:21.523394 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.523369 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:21.523394 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.523392 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:21.523394 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.523400 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:21.523587 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.523408 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:21.523587 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.523417 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:21.523587 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:21.523427 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww8mc\" (UniqueName: \"kubernetes.io/projected/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7-kube-api-access-ww8mc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:50:22.150785 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:22.150748 2579 generic.go:356] "Generic (PLEG): container finished" podID="dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" containerID="b7e30300aead0f42a1f0fcb27dfa07c1c7f20f1ec395ecfd8687ba38f8e0e7a8" exitCode=0 Sep 19 10:50:22.150953 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:22.150796 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" Sep 19 10:50:22.150953 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:22.150797 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" event={"ID":"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7","Type":"ContainerDied","Data":"b7e30300aead0f42a1f0fcb27dfa07c1c7f20f1ec395ecfd8687ba38f8e0e7a8"} Sep 19 10:50:22.151089 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:22.150949 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg" event={"ID":"dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7","Type":"ContainerDied","Data":"0d94cff97ea38de027484c0358d5661338a7235e7c91ff0d263cc7df716d4b7e"} Sep 19 10:50:22.151089 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:22.150990 2579 scope.go:117] "RemoveContainer" containerID="b7e30300aead0f42a1f0fcb27dfa07c1c7f20f1ec395ecfd8687ba38f8e0e7a8" Sep 19 10:50:22.161374 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:22.161298 2579 scope.go:117] "RemoveContainer" containerID="88009ddc08b425e8f2632d3dfb39eb08a8309ec8d102b18ae3893566a2f6d879" Sep 19 10:50:22.171011 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:22.170990 2579 scope.go:117] "RemoveContainer" containerID="b7e30300aead0f42a1f0fcb27dfa07c1c7f20f1ec395ecfd8687ba38f8e0e7a8" Sep 19 10:50:22.171377 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:50:22.171352 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e30300aead0f42a1f0fcb27dfa07c1c7f20f1ec395ecfd8687ba38f8e0e7a8\": container with ID starting with b7e30300aead0f42a1f0fcb27dfa07c1c7f20f1ec395ecfd8687ba38f8e0e7a8 not found: ID does not exist" containerID="b7e30300aead0f42a1f0fcb27dfa07c1c7f20f1ec395ecfd8687ba38f8e0e7a8" Sep 19 10:50:22.171509 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:22.171387 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e30300aead0f42a1f0fcb27dfa07c1c7f20f1ec395ecfd8687ba38f8e0e7a8"} err="failed to get container status \"b7e30300aead0f42a1f0fcb27dfa07c1c7f20f1ec395ecfd8687ba38f8e0e7a8\": rpc error: code = NotFound desc = could not find container \"b7e30300aead0f42a1f0fcb27dfa07c1c7f20f1ec395ecfd8687ba38f8e0e7a8\": container with ID starting with b7e30300aead0f42a1f0fcb27dfa07c1c7f20f1ec395ecfd8687ba38f8e0e7a8 not found: ID does not exist" Sep 19 10:50:22.171509 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:22.171412 2579 scope.go:117] "RemoveContainer" containerID="88009ddc08b425e8f2632d3dfb39eb08a8309ec8d102b18ae3893566a2f6d879" Sep 19 10:50:22.171741 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:50:22.171702 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88009ddc08b425e8f2632d3dfb39eb08a8309ec8d102b18ae3893566a2f6d879\": container with ID starting with 88009ddc08b425e8f2632d3dfb39eb08a8309ec8d102b18ae3893566a2f6d879 not found: ID does not exist" containerID="88009ddc08b425e8f2632d3dfb39eb08a8309ec8d102b18ae3893566a2f6d879" Sep 19 10:50:22.171858 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:22.171744 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88009ddc08b425e8f2632d3dfb39eb08a8309ec8d102b18ae3893566a2f6d879"} err="failed to get container status \"88009ddc08b425e8f2632d3dfb39eb08a8309ec8d102b18ae3893566a2f6d879\": rpc error: code = NotFound desc = could not find container \"88009ddc08b425e8f2632d3dfb39eb08a8309ec8d102b18ae3893566a2f6d879\": container with ID starting with 88009ddc08b425e8f2632d3dfb39eb08a8309ec8d102b18ae3893566a2f6d879 not found: ID does not exist" Sep 19 10:50:22.173333 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:22.173314 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg"] Sep 19 10:50:22.177137 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:22.177118 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg"] Sep 19 10:50:22.196025 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:50:22.196002 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" path="/var/lib/kubelet/pods/dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7/volumes" Sep 19 10:51:12.741560 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.741469 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr"] Sep 19 10:51:12.742148 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742117 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" containerName="istio-proxy" Sep 19 10:51:12.742148 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742139 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" containerName="istio-proxy" Sep 19 10:51:12.742282 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742164 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" containerName="istio-proxy" Sep 19 10:51:12.742282 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742172 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" containerName="istio-proxy" Sep 19 10:51:12.742282 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742193 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4830ab34-a086-4653-9db3-6f63a1e7286b" containerName="payload-pre-processing" Sep 19 10:51:12.742282 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742202 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4830ab34-a086-4653-9db3-6f63a1e7286b" containerName="payload-pre-processing" Sep 19 10:51:12.742282 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742212 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d4859e-7650-4582-8f3f-81f4f4440661" containerName="revoke-keys" Sep 19 10:51:12.742282 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742220 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d4859e-7650-4582-8f3f-81f4f4440661" containerName="revoke-keys" Sep 19 10:51:12.742282 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742241 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee53b8b-464c-42b8-bea2-34e17646094f" containerName="payload-processing" Sep 19 10:51:12.742282 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742250 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee53b8b-464c-42b8-bea2-34e17646094f" containerName="payload-processing" Sep 19 10:51:12.742675 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742345 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4830ab34-a086-4653-9db3-6f63a1e7286b" containerName="payload-pre-processing" Sep 19 10:51:12.742675 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742357 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" containerName="istio-proxy" Sep 19 10:51:12.742675 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742369 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d4859e-7650-4582-8f3f-81f4f4440661" containerName="revoke-keys" Sep 19 10:51:12.742675 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.742379 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee53b8b-464c-42b8-bea2-34e17646094f" containerName="payload-processing" Sep 19 10:51:12.745363 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.745338 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.748049 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.748021 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-580e01\"/\"default-dockercfg-kml78\"" Sep 19 10:51:12.748970 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.748952 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-580e01\"/\"kube-root-ca.crt\"" Sep 19 10:51:12.750255 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.749261 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-580e01\"/\"openshift-service-ca.crt\"" Sep 19 10:51:12.750255 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.749545 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-580e01\"/\"distinct-w3-580e01-kserve-self-signed-certs\"" Sep 19 10:51:12.752869 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.752844 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr"] Sep 19 10:51:12.808029 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.807992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-home\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.808196 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.808042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-dshm\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.808196 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.808095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216cf07a-3add-4258-820b-1be053bd9a0f-tls-certs\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.808196 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.808160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-model-cache\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.808365 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.808219 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-tmp-dir\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.808365 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.808352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cpxr\" (UniqueName: \"kubernetes.io/projected/216cf07a-3add-4258-820b-1be053bd9a0f-kube-api-access-8cpxr\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.909001 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.908960 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cpxr\" (UniqueName: \"kubernetes.io/projected/216cf07a-3add-4258-820b-1be053bd9a0f-kube-api-access-8cpxr\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.909181 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.909032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-home\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.909181 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.909089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-dshm\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.909181 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.909131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216cf07a-3add-4258-820b-1be053bd9a0f-tls-certs\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.909336 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.909192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-model-cache\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.909336 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.909268 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-tmp-dir\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.909467 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.909442 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-home\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.909559 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.909537 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-model-cache\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.909619 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.909596 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-tmp-dir\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.911678 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.911645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216cf07a-3add-4258-820b-1be053bd9a0f-tls-certs\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.911873 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.911853 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-dshm\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:12.916468 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:12.916446 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cpxr\" (UniqueName: \"kubernetes.io/projected/216cf07a-3add-4258-820b-1be053bd9a0f-kube-api-access-8cpxr\") pod \"distinct-w3-580e01-kserve-7c5d94d8f-kw5zr\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:13.065898 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:13.065823 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:13.346954 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:13.346925 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr"] Sep 19 10:51:13.348410 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:51:13.348383 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod216cf07a_3add_4258_820b_1be053bd9a0f.slice/crio-c9a20348e4b81ff3fde0fad2754cd66f3370a74fe336563955c6afc80d2e33e1 WatchSource:0}: Error finding container c9a20348e4b81ff3fde0fad2754cd66f3370a74fe336563955c6afc80d2e33e1: Status 404 returned error can't find the container with id c9a20348e4b81ff3fde0fad2754cd66f3370a74fe336563955c6afc80d2e33e1 Sep 19 10:51:13.427042 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:13.427009 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" event={"ID":"216cf07a-3add-4258-820b-1be053bd9a0f","Type":"ContainerStarted","Data":"80392e04c8a94d8af21c82e356af7341db5c3d9fe78f1092ce69d6b7060f36e4"} Sep 19 10:51:13.427183 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:13.427084 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" event={"ID":"216cf07a-3add-4258-820b-1be053bd9a0f","Type":"ContainerStarted","Data":"c9a20348e4b81ff3fde0fad2754cd66f3370a74fe336563955c6afc80d2e33e1"} Sep 19 10:51:13.443740 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:13.443696 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" podStartSLOduration=1.443680791 podStartE2EDuration="1.443680791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:51:13.441847282 +0000 UTC m=+1005.354351999" watchObservedRunningTime="2026-09-19 10:51:13.443680791 +0000 UTC m=+1005.356185507" Sep 19 10:51:23.066436 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:23.066399 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:23.066436 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:23.066444 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:23.079180 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:23.079150 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:23.499501 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:23.499469 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:51:50.062205 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.062171 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k"] Sep 19 10:51:50.062947 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.062928 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe0f42d-85eb-4b35-ba0f-c5e3c000a5e7" containerName="istio-proxy" Sep 19 10:51:50.066530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.066507 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.069053 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.069034 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c3f23ba4-openshift-default-dockercfg-n5pvr\"" Sep 19 10:51:50.077994 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.077968 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k"] Sep 19 10:51:50.162918 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.162876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-istio-envoy\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.163130 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.163016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0215709b-3393-43de-bd07-91da2f6e0bef-istio-podinfo\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.163130 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.163102 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-istio-data\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.163262 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.163146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-workload-certs\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.163262 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.163231 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0215709b-3393-43de-bd07-91da2f6e0bef-istio-token\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.163378 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.163303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0215709b-3393-43de-bd07-91da2f6e0bef-istiod-ca-cert\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.163378 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.163344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-workload-socket\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.163463 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.163374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-credential-socket\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.163463 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.163434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmb8\" (UniqueName: \"kubernetes.io/projected/0215709b-3393-43de-bd07-91da2f6e0bef-kube-api-access-xnmb8\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.264910 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.264877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0215709b-3393-43de-bd07-91da2f6e0bef-istiod-ca-cert\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.264910 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.264910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-workload-socket\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.265203 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.264930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-credential-socket\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.265203 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.264957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmb8\" (UniqueName: \"kubernetes.io/projected/0215709b-3393-43de-bd07-91da2f6e0bef-kube-api-access-xnmb8\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.265203 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.265022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-istio-envoy\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.265203 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.265130 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0215709b-3393-43de-bd07-91da2f6e0bef-istio-podinfo\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.265203 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.265169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-istio-data\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.265468 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.265213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-workload-certs\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.265468 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.265260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0215709b-3393-43de-bd07-91da2f6e0bef-istio-token\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.265577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.265515 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-workload-socket\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.265577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.265559 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-credential-socket\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.265700 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.265680 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-istio-data\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.265762 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.265735 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-workload-certs\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.265907 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.265887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0215709b-3393-43de-bd07-91da2f6e0bef-istiod-ca-cert\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.267786 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.267762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-istio-envoy\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.268130 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.268108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0215709b-3393-43de-bd07-91da2f6e0bef-istio-podinfo\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.272925 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.272902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0215709b-3393-43de-bd07-91da2f6e0bef-istio-token\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.273370 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.273348 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmb8\" (UniqueName: \"kubernetes.io/projected/0215709b-3393-43de-bd07-91da2f6e0bef-kube-api-access-xnmb8\") pod \"e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.378838 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.378764 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:50.652573 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.652541 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k"] Sep 19 10:51:50.654154 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:51:50.654126 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0215709b_3393_43de_bd07_91da2f6e0bef.slice/crio-e38f9a1d80ef4119c3ecfbeb5ef709a414c2d6793e165d22c7ffbf857b00fc6a WatchSource:0}: Error finding container e38f9a1d80ef4119c3ecfbeb5ef709a414c2d6793e165d22c7ffbf857b00fc6a: Status 404 returned error can't find the container with id e38f9a1d80ef4119c3ecfbeb5ef709a414c2d6793e165d22c7ffbf857b00fc6a Sep 19 10:51:50.656182 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.656153 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:51:50.656247 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.656214 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:51:50.656247 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:50.656242 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:51:51.640650 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:51.640617 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" event={"ID":"0215709b-3393-43de-bd07-91da2f6e0bef","Type":"ContainerStarted","Data":"d770f1dbd0d7679c50362bafb9ee09c666c7c73f0e8f63f12bf8288d4b2f786e"} Sep 19 10:51:51.641031 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:51.640697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" event={"ID":"0215709b-3393-43de-bd07-91da2f6e0bef","Type":"ContainerStarted","Data":"e38f9a1d80ef4119c3ecfbeb5ef709a414c2d6793e165d22c7ffbf857b00fc6a"} Sep 19 10:51:51.660159 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:51.660112 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" podStartSLOduration=1.660096072 podStartE2EDuration="1.660096072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:51:51.658967995 +0000 UTC m=+1043.571472712" watchObservedRunningTime="2026-09-19 10:51:51.660096072 +0000 UTC m=+1043.572600785" Sep 19 10:51:52.379705 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:52.379673 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:52.384833 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:52.384807 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:52.646469 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:52.646446 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:52.647419 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:52.647399 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:51:56.917029 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:56.916989 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb"] Sep 19 10:51:56.920854 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:56.920837 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" Sep 19 10:51:56.923443 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:56.923419 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c3f23ba4-dockercfg-kg6xk\"" Sep 19 10:51:56.923570 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:56.923478 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c3f23ba4\"" Sep 19 10:51:56.932502 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:56.932465 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb"] Sep 19 10:51:56.943155 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:56.943119 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn"] Sep 19 10:51:56.946541 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:56.946519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" Sep 19 10:51:56.952562 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:56.952540 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn"] Sep 19 10:51:57.036849 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.036814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wqwv\" (UniqueName: \"kubernetes.io/projected/d0a9ac77-d847-4241-b9a7-b935d6855db8-kube-api-access-5wqwv\") pod \"payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb\" (UID: \"d0a9ac77-d847-4241-b9a7-b935d6855db8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" Sep 19 10:51:57.037049 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.036889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a9ac77-d847-4241-b9a7-b935d6855db8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb\" (UID: \"d0a9ac77-d847-4241-b9a7-b935d6855db8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" Sep 19 10:51:57.037049 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.037003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d75e21-0a89-4c94-8cba-030de1f94dbf-plugins-config-volume\") pod \"payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn\" (UID: \"99d75e21-0a89-4c94-8cba-030de1f94dbf\") " pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" Sep 19 10:51:57.037215 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.037105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-478xj\" (UniqueName: \"kubernetes.io/projected/99d75e21-0a89-4c94-8cba-030de1f94dbf-kube-api-access-478xj\") pod \"payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn\" (UID: \"99d75e21-0a89-4c94-8cba-030de1f94dbf\") " pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" Sep 19 10:51:57.137977 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.137948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a9ac77-d847-4241-b9a7-b935d6855db8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb\" (UID: \"d0a9ac77-d847-4241-b9a7-b935d6855db8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" Sep 19 10:51:57.138182 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.138006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d75e21-0a89-4c94-8cba-030de1f94dbf-plugins-config-volume\") pod \"payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn\" (UID: \"99d75e21-0a89-4c94-8cba-030de1f94dbf\") " pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" Sep 19 10:51:57.138182 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.138087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-478xj\" (UniqueName: \"kubernetes.io/projected/99d75e21-0a89-4c94-8cba-030de1f94dbf-kube-api-access-478xj\") pod \"payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn\" (UID: \"99d75e21-0a89-4c94-8cba-030de1f94dbf\") " pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" Sep 19 10:51:57.138182 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.138110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wqwv\" (UniqueName: \"kubernetes.io/projected/d0a9ac77-d847-4241-b9a7-b935d6855db8-kube-api-access-5wqwv\") pod \"payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb\" (UID: \"d0a9ac77-d847-4241-b9a7-b935d6855db8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" Sep 19 10:51:57.138774 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.138747 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d75e21-0a89-4c94-8cba-030de1f94dbf-plugins-config-volume\") pod \"payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn\" (UID: \"99d75e21-0a89-4c94-8cba-030de1f94dbf\") " pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" Sep 19 10:51:57.138893 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.138875 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a9ac77-d847-4241-b9a7-b935d6855db8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb\" (UID: \"d0a9ac77-d847-4241-b9a7-b935d6855db8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" Sep 19 10:51:57.145778 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.145751 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-478xj\" (UniqueName: \"kubernetes.io/projected/99d75e21-0a89-4c94-8cba-030de1f94dbf-kube-api-access-478xj\") pod \"payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn\" (UID: \"99d75e21-0a89-4c94-8cba-030de1f94dbf\") " pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" Sep 19 10:51:57.145898 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.145765 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wqwv\" (UniqueName: \"kubernetes.io/projected/d0a9ac77-d847-4241-b9a7-b935d6855db8-kube-api-access-5wqwv\") pod \"payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb\" (UID: \"d0a9ac77-d847-4241-b9a7-b935d6855db8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" Sep 19 10:51:57.241038 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.240947 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" Sep 19 10:51:57.258336 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.258302 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" Sep 19 10:51:57.519283 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.519256 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb"] Sep 19 10:51:57.520816 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:51:57.520783 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a9ac77_d847_4241_b9a7_b935d6855db8.slice/crio-bd10fea51b562e4d795242e5ef6b5d8f3f37c58720932c695efe138c63ecfefe WatchSource:0}: Error finding container bd10fea51b562e4d795242e5ef6b5d8f3f37c58720932c695efe138c63ecfefe: Status 404 returned error can't find the container with id bd10fea51b562e4d795242e5ef6b5d8f3f37c58720932c695efe138c63ecfefe Sep 19 10:51:57.547358 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.547331 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn"] Sep 19 10:51:57.550210 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:51:57.550183 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d75e21_0a89_4c94_8cba_030de1f94dbf.slice/crio-f05c93e416ab8f06031f9f6d06363af0dbaae91c7671a61dbe67b93bd048d555 WatchSource:0}: Error finding container f05c93e416ab8f06031f9f6d06363af0dbaae91c7671a61dbe67b93bd048d555: Status 404 returned error can't find the container with id f05c93e416ab8f06031f9f6d06363af0dbaae91c7671a61dbe67b93bd048d555 Sep 19 10:51:57.676426 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.676385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" event={"ID":"d0a9ac77-d847-4241-b9a7-b935d6855db8","Type":"ContainerStarted","Data":"f0d17f243d67c25318926752fb2e9a57a852e2ab93f547517abfc02346100e97"} Sep 19 10:51:57.676612 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.676450 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" event={"ID":"d0a9ac77-d847-4241-b9a7-b935d6855db8","Type":"ContainerStarted","Data":"bd10fea51b562e4d795242e5ef6b5d8f3f37c58720932c695efe138c63ecfefe"} Sep 19 10:51:57.676612 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.676511 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" Sep 19 10:51:57.678095 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.678053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" event={"ID":"99d75e21-0a89-4c94-8cba-030de1f94dbf","Type":"ContainerStarted","Data":"8c2a4aaa135da8c080da43f67b94064886c5ec22246c862e0b41d338ca200f9c"} Sep 19 10:51:57.678212 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.678115 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" event={"ID":"99d75e21-0a89-4c94-8cba-030de1f94dbf","Type":"ContainerStarted","Data":"f05c93e416ab8f06031f9f6d06363af0dbaae91c7671a61dbe67b93bd048d555"} Sep 19 10:51:57.678275 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.678207 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" Sep 19 10:51:57.692632 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.692589 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" podStartSLOduration=1.692572179 podStartE2EDuration="1.692572179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:51:57.69065031 +0000 UTC m=+1049.603155032" watchObservedRunningTime="2026-09-19 10:51:57.692572179 +0000 UTC m=+1049.605076896" Sep 19 10:51:57.707159 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:51:57.707113 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" podStartSLOduration=1.707097004 podStartE2EDuration="1.707097004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:51:57.704499933 +0000 UTC m=+1049.617004644" watchObservedRunningTime="2026-09-19 10:51:57.707097004 +0000 UTC m=+1049.619601724" Sep 19 10:52:08.686222 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:08.686189 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" Sep 19 10:52:08.686615 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:08.686257 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" Sep 19 10:52:23.647471 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:23.647424 2579 patch_prober.go:28] interesting pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 19 10:52:23.647948 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:23.647502 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 19 10:52:25.833672 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:25.833634 2579 generic.go:356] "Generic (PLEG): container finished" podID="0215709b-3393-43de-bd07-91da2f6e0bef" containerID="d770f1dbd0d7679c50362bafb9ee09c666c7c73f0e8f63f12bf8288d4b2f786e" exitCode=0 Sep 19 10:52:25.834091 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:25.833704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" event={"ID":"0215709b-3393-43de-bd07-91da2f6e0bef","Type":"ContainerDied","Data":"d770f1dbd0d7679c50362bafb9ee09c666c7c73f0e8f63f12bf8288d4b2f786e"} Sep 19 10:52:25.834091 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:25.834086 2579 scope.go:117] "RemoveContainer" containerID="d770f1dbd0d7679c50362bafb9ee09c666c7c73f0e8f63f12bf8288d4b2f786e" Sep 19 10:52:25.835347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:25.835323 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:52:25.835408 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:25.835389 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:52:25.835445 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:25.835419 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:52:26.843230 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:26.843184 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" event={"ID":"0215709b-3393-43de-bd07-91da2f6e0bef","Type":"ContainerStarted","Data":"ed8c6ed6d617164b49f6794e86f37877ea30b431d94ff050171aad61a54a12dd"} Sep 19 10:52:27.379685 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:27.379658 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:52:27.379855 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:27.379833 2579 patch_prober.go:28] interesting pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 19 10:52:27.379903 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:27.379876 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 19 10:52:28.379307 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:28.379274 2579 patch_prober.go:28] interesting pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 19 10:52:28.379740 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:28.379340 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 19 10:52:29.379630 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:29.379595 2579 patch_prober.go:28] interesting pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 19 10:52:29.380008 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:29.379651 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 19 10:52:30.379787 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:30.379751 2579 patch_prober.go:28] interesting pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 19 10:52:30.380202 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:30.379832 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 19 10:52:30.868377 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:30.868340 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7"] Sep 19 10:52:30.872771 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:30.872741 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7" Sep 19 10:52:30.875763 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:30.875501 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 10:52:30.875763 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:30.875501 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 10:52:30.882720 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:30.882615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7"] Sep 19 10:52:30.995647 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:30.995599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bd42e0-d043-4b72-9d12-f9b70ec67fac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7\" (UID: \"f8bd42e0-d043-4b72-9d12-f9b70ec67fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7" Sep 19 10:52:31.096962 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:31.096928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bd42e0-d043-4b72-9d12-f9b70ec67fac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7\" (UID: \"f8bd42e0-d043-4b72-9d12-f9b70ec67fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7" Sep 19 10:52:31.097578 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:31.097555 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bd42e0-d043-4b72-9d12-f9b70ec67fac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7\" (UID: \"f8bd42e0-d043-4b72-9d12-f9b70ec67fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7" Sep 19 10:52:31.184628 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:31.184595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7" Sep 19 10:52:31.379739 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:31.379709 2579 patch_prober.go:28] interesting pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 19 10:52:31.379900 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:31.379764 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 19 10:52:31.484742 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:31.484658 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7"] Sep 19 10:52:31.491591 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:52:31.491553 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bd42e0_d043_4b72_9d12_f9b70ec67fac.slice/crio-f9b687e2b325cb9ebae503cb7cc74d1d4b554a3c5b92c2de712d8ed5b6b586db WatchSource:0}: Error finding container f9b687e2b325cb9ebae503cb7cc74d1d4b554a3c5b92c2de712d8ed5b6b586db: Status 404 returned error can't find the container with id f9b687e2b325cb9ebae503cb7cc74d1d4b554a3c5b92c2de712d8ed5b6b586db Sep 19 10:52:31.885404 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:31.885367 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7" event={"ID":"f8bd42e0-d043-4b72-9d12-f9b70ec67fac","Type":"ContainerStarted","Data":"4f243e3969fedd3474e16be408430bd3d7d818dca9e70e2751b5f7c1a3361039"} Sep 19 10:52:31.885566 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:31.885429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7" event={"ID":"f8bd42e0-d043-4b72-9d12-f9b70ec67fac","Type":"ContainerStarted","Data":"f9b687e2b325cb9ebae503cb7cc74d1d4b554a3c5b92c2de712d8ed5b6b586db"} Sep 19 10:52:31.901515 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:31.901466 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7" podStartSLOduration=1.901450319 podStartE2EDuration="1.901450319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:52:31.900333038 +0000 UTC m=+1083.812837755" watchObservedRunningTime="2026-09-19 10:52:31.901450319 +0000 UTC m=+1083.813955072" Sep 19 10:52:32.382679 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:32.382647 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:52:32.383045 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:32.382923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:52:32.383532 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:32.383511 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:52:32.892471 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:32.892441 2579 generic.go:356] "Generic (PLEG): container finished" podID="f8bd42e0-d043-4b72-9d12-f9b70ec67fac" containerID="4f243e3969fedd3474e16be408430bd3d7d818dca9e70e2751b5f7c1a3361039" exitCode=0 Sep 19 10:52:32.892631 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:32.892510 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7" event={"ID":"f8bd42e0-d043-4b72-9d12-f9b70ec67fac","Type":"ContainerDied","Data":"4f243e3969fedd3474e16be408430bd3d7d818dca9e70e2751b5f7c1a3361039"} Sep 19 10:52:34.034730 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.034704 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7" Sep 19 10:52:34.228846 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.228764 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bd42e0-d043-4b72-9d12-f9b70ec67fac-maas-api-service-ca\") pod \"f8bd42e0-d043-4b72-9d12-f9b70ec67fac\" (UID: \"f8bd42e0-d043-4b72-9d12-f9b70ec67fac\") " Sep 19 10:52:34.229118 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.229091 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bd42e0-d043-4b72-9d12-f9b70ec67fac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8bd42e0-d043-4b72-9d12-f9b70ec67fac" (UID: "f8bd42e0-d043-4b72-9d12-f9b70ec67fac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:52:34.330152 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.330112 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bd42e0-d043-4b72-9d12-f9b70ec67fac-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:34.340229 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.340195 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp"] Sep 19 10:52:34.340645 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.340632 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8bd42e0-d043-4b72-9d12-f9b70ec67fac" containerName="revoke-keys" Sep 19 10:52:34.340702 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.340646 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8bd42e0-d043-4b72-9d12-f9b70ec67fac" containerName="revoke-keys" Sep 19 10:52:34.340745 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.340725 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8bd42e0-d043-4b72-9d12-f9b70ec67fac" containerName="revoke-keys" Sep 19 10:52:34.343994 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.343977 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp" Sep 19 10:52:34.351512 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.351481 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp"] Sep 19 10:52:34.532423 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.532328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e771cd-9cb7-4c32-ac68-11c8ffb8f03c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp\" (UID: \"64e771cd-9cb7-4c32-ac68-11c8ffb8f03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp" Sep 19 10:52:34.633477 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.633447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e771cd-9cb7-4c32-ac68-11c8ffb8f03c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp\" (UID: \"64e771cd-9cb7-4c32-ac68-11c8ffb8f03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp" Sep 19 10:52:34.634112 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.634088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e771cd-9cb7-4c32-ac68-11c8ffb8f03c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp\" (UID: \"64e771cd-9cb7-4c32-ac68-11c8ffb8f03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp" Sep 19 10:52:34.656748 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.656728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp" Sep 19 10:52:34.903769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.903738 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7" Sep 19 10:52:34.903769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.903762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7" event={"ID":"f8bd42e0-d043-4b72-9d12-f9b70ec67fac","Type":"ContainerDied","Data":"f9b687e2b325cb9ebae503cb7cc74d1d4b554a3c5b92c2de712d8ed5b6b586db"} Sep 19 10:52:34.903974 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.903811 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9b687e2b325cb9ebae503cb7cc74d1d4b554a3c5b92c2de712d8ed5b6b586db" Sep 19 10:52:34.927169 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:34.927147 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp"] Sep 19 10:52:34.929676 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:52:34.929649 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e771cd_9cb7_4c32_ac68_11c8ffb8f03c.slice/crio-f1cca5926441f2dd94ac0909061f52572fac1dd982c8e8b2e5ec5ccea4a4d8d6 WatchSource:0}: Error finding container f1cca5926441f2dd94ac0909061f52572fac1dd982c8e8b2e5ec5ccea4a4d8d6: Status 404 returned error can't find the container with id f1cca5926441f2dd94ac0909061f52572fac1dd982c8e8b2e5ec5ccea4a4d8d6 Sep 19 10:52:35.910501 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:35.910414 2579 generic.go:356] "Generic (PLEG): container finished" podID="64e771cd-9cb7-4c32-ac68-11c8ffb8f03c" containerID="36430b7b10657a3efee431b47a7ee8239cee26a0e6bb6bfb5c05087f799e7e06" exitCode=0 Sep 19 10:52:35.910865 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:35.910498 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp" event={"ID":"64e771cd-9cb7-4c32-ac68-11c8ffb8f03c","Type":"ContainerDied","Data":"36430b7b10657a3efee431b47a7ee8239cee26a0e6bb6bfb5c05087f799e7e06"} Sep 19 10:52:35.910865 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:35.910575 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp" event={"ID":"64e771cd-9cb7-4c32-ac68-11c8ffb8f03c","Type":"ContainerStarted","Data":"f1cca5926441f2dd94ac0909061f52572fac1dd982c8e8b2e5ec5ccea4a4d8d6"} Sep 19 10:52:37.055399 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:37.055370 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp" Sep 19 10:52:37.062486 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:37.062464 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e771cd-9cb7-4c32-ac68-11c8ffb8f03c-maas-api-service-ca\") pod \"64e771cd-9cb7-4c32-ac68-11c8ffb8f03c\" (UID: \"64e771cd-9cb7-4c32-ac68-11c8ffb8f03c\") " Sep 19 10:52:37.062771 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:37.062751 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e771cd-9cb7-4c32-ac68-11c8ffb8f03c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64e771cd-9cb7-4c32-ac68-11c8ffb8f03c" (UID: "64e771cd-9cb7-4c32-ac68-11c8ffb8f03c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:52:37.163408 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:37.163383 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e771cd-9cb7-4c32-ac68-11c8ffb8f03c-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:37.923536 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:37.923509 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp" Sep 19 10:52:37.923708 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:37.923509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp" event={"ID":"64e771cd-9cb7-4c32-ac68-11c8ffb8f03c","Type":"ContainerDied","Data":"f1cca5926441f2dd94ac0909061f52572fac1dd982c8e8b2e5ec5ccea4a4d8d6"} Sep 19 10:52:37.923708 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:37.923632 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1cca5926441f2dd94ac0909061f52572fac1dd982c8e8b2e5ec5ccea4a4d8d6" Sep 19 10:52:41.571859 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.571826 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn"] Sep 19 10:52:41.572293 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.572073 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" podUID="99d75e21-0a89-4c94-8cba-030de1f94dbf" containerName="payload-processing" containerID="cri-o://8c2a4aaa135da8c080da43f67b94064886c5ec22246c862e0b41d338ca200f9c" gracePeriod=30 Sep 19 10:52:41.579535 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.579509 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb"] Sep 19 10:52:41.580556 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.580512 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" podUID="d0a9ac77-d847-4241-b9a7-b935d6855db8" containerName="payload-pre-processing" containerID="cri-o://f0d17f243d67c25318926752fb2e9a57a852e2ab93f547517abfc02346100e97" gracePeriod=30 Sep 19 10:52:41.718212 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.718180 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7"] Sep 19 10:52:41.720612 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.720573 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3f23ba4-48hr7"] Sep 19 10:52:41.862991 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.862967 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" Sep 19 10:52:41.867451 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.867431 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" Sep 19 10:52:41.907844 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.907821 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wqwv\" (UniqueName: \"kubernetes.io/projected/d0a9ac77-d847-4241-b9a7-b935d6855db8-kube-api-access-5wqwv\") pod \"d0a9ac77-d847-4241-b9a7-b935d6855db8\" (UID: \"d0a9ac77-d847-4241-b9a7-b935d6855db8\") " Sep 19 10:52:41.907968 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.907948 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a9ac77-d847-4241-b9a7-b935d6855db8-plugins-config-volume\") pod \"d0a9ac77-d847-4241-b9a7-b935d6855db8\" (UID: \"d0a9ac77-d847-4241-b9a7-b935d6855db8\") " Sep 19 10:52:41.908021 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.908008 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-478xj\" (UniqueName: \"kubernetes.io/projected/99d75e21-0a89-4c94-8cba-030de1f94dbf-kube-api-access-478xj\") pod \"99d75e21-0a89-4c94-8cba-030de1f94dbf\" (UID: \"99d75e21-0a89-4c94-8cba-030de1f94dbf\") " Sep 19 10:52:41.908157 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.908108 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d75e21-0a89-4c94-8cba-030de1f94dbf-plugins-config-volume\") pod \"99d75e21-0a89-4c94-8cba-030de1f94dbf\" (UID: \"99d75e21-0a89-4c94-8cba-030de1f94dbf\") " Sep 19 10:52:41.908527 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.908438 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a9ac77-d847-4241-b9a7-b935d6855db8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0a9ac77-d847-4241-b9a7-b935d6855db8" (UID: "d0a9ac77-d847-4241-b9a7-b935d6855db8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:52:41.908527 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.908503 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d75e21-0a89-4c94-8cba-030de1f94dbf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99d75e21-0a89-4c94-8cba-030de1f94dbf" (UID: "99d75e21-0a89-4c94-8cba-030de1f94dbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:52:41.910420 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.910372 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d75e21-0a89-4c94-8cba-030de1f94dbf-kube-api-access-478xj" (OuterVolumeSpecName: "kube-api-access-478xj") pod "99d75e21-0a89-4c94-8cba-030de1f94dbf" (UID: "99d75e21-0a89-4c94-8cba-030de1f94dbf"). InnerVolumeSpecName "kube-api-access-478xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:52:41.935196 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.917436 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a9ac77-d847-4241-b9a7-b935d6855db8-kube-api-access-5wqwv" (OuterVolumeSpecName: "kube-api-access-5wqwv") pod "d0a9ac77-d847-4241-b9a7-b935d6855db8" (UID: "d0a9ac77-d847-4241-b9a7-b935d6855db8"). InnerVolumeSpecName "kube-api-access-5wqwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:52:41.957700 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.957663 2579 generic.go:356] "Generic (PLEG): container finished" podID="d0a9ac77-d847-4241-b9a7-b935d6855db8" containerID="f0d17f243d67c25318926752fb2e9a57a852e2ab93f547517abfc02346100e97" exitCode=0 Sep 19 10:52:41.957898 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.957717 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" Sep 19 10:52:41.957898 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.957764 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" event={"ID":"d0a9ac77-d847-4241-b9a7-b935d6855db8","Type":"ContainerDied","Data":"f0d17f243d67c25318926752fb2e9a57a852e2ab93f547517abfc02346100e97"} Sep 19 10:52:41.957898 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.957812 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb" event={"ID":"d0a9ac77-d847-4241-b9a7-b935d6855db8","Type":"ContainerDied","Data":"bd10fea51b562e4d795242e5ef6b5d8f3f37c58720932c695efe138c63ecfefe"} Sep 19 10:52:41.957898 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.957838 2579 scope.go:117] "RemoveContainer" containerID="f0d17f243d67c25318926752fb2e9a57a852e2ab93f547517abfc02346100e97" Sep 19 10:52:41.959593 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.959466 2579 generic.go:356] "Generic (PLEG): container finished" podID="99d75e21-0a89-4c94-8cba-030de1f94dbf" containerID="8c2a4aaa135da8c080da43f67b94064886c5ec22246c862e0b41d338ca200f9c" exitCode=0 Sep 19 10:52:41.959593 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.959502 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" Sep 19 10:52:41.959593 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.959521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" event={"ID":"99d75e21-0a89-4c94-8cba-030de1f94dbf","Type":"ContainerDied","Data":"8c2a4aaa135da8c080da43f67b94064886c5ec22246c862e0b41d338ca200f9c"} Sep 19 10:52:41.959593 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.959563 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn" event={"ID":"99d75e21-0a89-4c94-8cba-030de1f94dbf","Type":"ContainerDied","Data":"f05c93e416ab8f06031f9f6d06363af0dbaae91c7671a61dbe67b93bd048d555"} Sep 19 10:52:41.969533 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.969516 2579 scope.go:117] "RemoveContainer" containerID="f0d17f243d67c25318926752fb2e9a57a852e2ab93f547517abfc02346100e97" Sep 19 10:52:41.969837 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:52:41.969816 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d17f243d67c25318926752fb2e9a57a852e2ab93f547517abfc02346100e97\": container with ID starting with f0d17f243d67c25318926752fb2e9a57a852e2ab93f547517abfc02346100e97 not found: ID does not exist" containerID="f0d17f243d67c25318926752fb2e9a57a852e2ab93f547517abfc02346100e97" Sep 19 10:52:41.969925 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.969848 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d17f243d67c25318926752fb2e9a57a852e2ab93f547517abfc02346100e97"} err="failed to get container status \"f0d17f243d67c25318926752fb2e9a57a852e2ab93f547517abfc02346100e97\": rpc error: code = NotFound desc = could not find container \"f0d17f243d67c25318926752fb2e9a57a852e2ab93f547517abfc02346100e97\": container with ID starting with f0d17f243d67c25318926752fb2e9a57a852e2ab93f547517abfc02346100e97 not found: ID does not exist" Sep 19 10:52:41.969925 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.969871 2579 scope.go:117] "RemoveContainer" containerID="8c2a4aaa135da8c080da43f67b94064886c5ec22246c862e0b41d338ca200f9c" Sep 19 10:52:41.978548 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.978521 2579 scope.go:117] "RemoveContainer" containerID="8c2a4aaa135da8c080da43f67b94064886c5ec22246c862e0b41d338ca200f9c" Sep 19 10:52:41.978854 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:52:41.978835 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2a4aaa135da8c080da43f67b94064886c5ec22246c862e0b41d338ca200f9c\": container with ID starting with 8c2a4aaa135da8c080da43f67b94064886c5ec22246c862e0b41d338ca200f9c not found: ID does not exist" containerID="8c2a4aaa135da8c080da43f67b94064886c5ec22246c862e0b41d338ca200f9c" Sep 19 10:52:41.978923 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.978865 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2a4aaa135da8c080da43f67b94064886c5ec22246c862e0b41d338ca200f9c"} err="failed to get container status \"8c2a4aaa135da8c080da43f67b94064886c5ec22246c862e0b41d338ca200f9c\": rpc error: code = NotFound desc = could not find container \"8c2a4aaa135da8c080da43f67b94064886c5ec22246c862e0b41d338ca200f9c\": container with ID starting with 8c2a4aaa135da8c080da43f67b94064886c5ec22246c862e0b41d338ca200f9c not found: ID does not exist" Sep 19 10:52:41.986012 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.985986 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb"] Sep 19 10:52:41.989402 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.989371 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb"] Sep 19 10:52:41.996409 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:41.996383 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn"] Sep 19 10:52:42.000382 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:42.000359 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn"] Sep 19 10:52:42.009631 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:42.009605 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a9ac77-d847-4241-b9a7-b935d6855db8-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:42.009631 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:42.009627 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-478xj\" (UniqueName: \"kubernetes.io/projected/99d75e21-0a89-4c94-8cba-030de1f94dbf-kube-api-access-478xj\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:42.009773 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:42.009639 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d75e21-0a89-4c94-8cba-030de1f94dbf-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:42.009773 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:42.009648 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wqwv\" (UniqueName: \"kubernetes.io/projected/d0a9ac77-d847-4241-b9a7-b935d6855db8-kube-api-access-5wqwv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:42.196690 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:42.196661 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99d75e21-0a89-4c94-8cba-030de1f94dbf" path="/var/lib/kubelet/pods/99d75e21-0a89-4c94-8cba-030de1f94dbf/volumes" Sep 19 10:52:42.197087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:42.197072 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0a9ac77-d847-4241-b9a7-b935d6855db8" path="/var/lib/kubelet/pods/d0a9ac77-d847-4241-b9a7-b935d6855db8/volumes" Sep 19 10:52:42.197376 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:42.197364 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8bd42e0-d043-4b72-9d12-f9b70ec67fac" path="/var/lib/kubelet/pods/f8bd42e0-d043-4b72-9d12-f9b70ec67fac/volumes" Sep 19 10:52:45.433389 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.433356 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb"] Sep 19 10:52:45.433803 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.433564 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" podUID="1d2dd5b8-8563-4db8-b1e0-6fedc133fbee" containerName="payload-pre-processing" containerID="cri-o://196eb7a2b4f079ae556673d046ec6932c105bf0e217dd4c2d64cc553abd81744" gracePeriod=30 Sep 19 10:52:45.577759 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.577719 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp"] Sep 19 10:52:45.581185 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.581153 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f91f97-zp6vp"] Sep 19 10:52:45.717941 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.717909 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" Sep 19 10:52:45.749192 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.749161 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2dd5b8-8563-4db8-b1e0-6fedc133fbee-plugins-config-volume\") pod \"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee\" (UID: \"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee\") " Sep 19 10:52:45.749377 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.749307 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-465sp\" (UniqueName: \"kubernetes.io/projected/1d2dd5b8-8563-4db8-b1e0-6fedc133fbee-kube-api-access-465sp\") pod \"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee\" (UID: \"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee\") " Sep 19 10:52:45.749568 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.749546 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d2dd5b8-8563-4db8-b1e0-6fedc133fbee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d2dd5b8-8563-4db8-b1e0-6fedc133fbee" (UID: "1d2dd5b8-8563-4db8-b1e0-6fedc133fbee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:52:45.749667 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.749649 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2dd5b8-8563-4db8-b1e0-6fedc133fbee-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:45.751474 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.751447 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2dd5b8-8563-4db8-b1e0-6fedc133fbee-kube-api-access-465sp" (OuterVolumeSpecName: "kube-api-access-465sp") pod "1d2dd5b8-8563-4db8-b1e0-6fedc133fbee" (UID: "1d2dd5b8-8563-4db8-b1e0-6fedc133fbee"). InnerVolumeSpecName "kube-api-access-465sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:52:45.850664 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.850633 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-465sp\" (UniqueName: \"kubernetes.io/projected/1d2dd5b8-8563-4db8-b1e0-6fedc133fbee-kube-api-access-465sp\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:45.989757 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.989670 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d2dd5b8-8563-4db8-b1e0-6fedc133fbee" containerID="196eb7a2b4f079ae556673d046ec6932c105bf0e217dd4c2d64cc553abd81744" exitCode=0 Sep 19 10:52:45.989757 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.989716 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" Sep 19 10:52:45.989962 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.989756 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" event={"ID":"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee","Type":"ContainerDied","Data":"196eb7a2b4f079ae556673d046ec6932c105bf0e217dd4c2d64cc553abd81744"} Sep 19 10:52:45.989962 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.989812 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb" event={"ID":"1d2dd5b8-8563-4db8-b1e0-6fedc133fbee","Type":"ContainerDied","Data":"451ebd2752bc4073448614fe01a4db81d49d694f648424c4321480b47fd36ffc"} Sep 19 10:52:45.989962 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.989841 2579 scope.go:117] "RemoveContainer" containerID="196eb7a2b4f079ae556673d046ec6932c105bf0e217dd4c2d64cc553abd81744" Sep 19 10:52:45.999053 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.999036 2579 scope.go:117] "RemoveContainer" containerID="196eb7a2b4f079ae556673d046ec6932c105bf0e217dd4c2d64cc553abd81744" Sep 19 10:52:45.999309 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:52:45.999289 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196eb7a2b4f079ae556673d046ec6932c105bf0e217dd4c2d64cc553abd81744\": container with ID starting with 196eb7a2b4f079ae556673d046ec6932c105bf0e217dd4c2d64cc553abd81744 not found: ID does not exist" containerID="196eb7a2b4f079ae556673d046ec6932c105bf0e217dd4c2d64cc553abd81744" Sep 19 10:52:45.999363 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:45.999316 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196eb7a2b4f079ae556673d046ec6932c105bf0e217dd4c2d64cc553abd81744"} err="failed to get container status \"196eb7a2b4f079ae556673d046ec6932c105bf0e217dd4c2d64cc553abd81744\": rpc error: code = NotFound desc = could not find container \"196eb7a2b4f079ae556673d046ec6932c105bf0e217dd4c2d64cc553abd81744\": container with ID starting with 196eb7a2b4f079ae556673d046ec6932c105bf0e217dd4c2d64cc553abd81744 not found: ID does not exist" Sep 19 10:52:46.010783 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:46.010761 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb"] Sep 19 10:52:46.012947 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:46.012928 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb"] Sep 19 10:52:46.196614 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:46.196584 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d2dd5b8-8563-4db8-b1e0-6fedc133fbee" path="/var/lib/kubelet/pods/1d2dd5b8-8563-4db8-b1e0-6fedc133fbee/volumes" Sep 19 10:52:46.196940 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:46.196928 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64e771cd-9cb7-4c32-ac68-11c8ffb8f03c" path="/var/lib/kubelet/pods/64e771cd-9cb7-4c32-ac68-11c8ffb8f03c/volumes" Sep 19 10:52:48.526024 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:48.525992 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k"] Sep 19 10:52:48.526565 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:48.526279 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" containerName="istio-proxy" containerID="cri-o://ed8c6ed6d617164b49f6794e86f37877ea30b431d94ff050171aad61a54a12dd" gracePeriod=30 Sep 19 10:52:51.817723 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:51.817687 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8"] Sep 19 10:52:51.818128 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:51.817971 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" containerID="cri-o://575e52a79f1f70337b626cf28f6869f08a8bc46d6d5b8e4b849822cc274e5ae2" gracePeriod=30 Sep 19 10:52:52.140285 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.140255 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2"] Sep 19 10:52:52.140682 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.140670 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d2dd5b8-8563-4db8-b1e0-6fedc133fbee" containerName="payload-pre-processing" Sep 19 10:52:52.140737 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.140683 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d2dd5b8-8563-4db8-b1e0-6fedc133fbee" containerName="payload-pre-processing" Sep 19 10:52:52.140737 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.140693 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a9ac77-d847-4241-b9a7-b935d6855db8" containerName="payload-pre-processing" Sep 19 10:52:52.140737 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.140699 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a9ac77-d847-4241-b9a7-b935d6855db8" containerName="payload-pre-processing" Sep 19 10:52:52.140737 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.140723 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d75e21-0a89-4c94-8cba-030de1f94dbf" containerName="payload-processing" Sep 19 10:52:52.140737 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.140729 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d75e21-0a89-4c94-8cba-030de1f94dbf" containerName="payload-processing" Sep 19 10:52:52.140737 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.140737 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64e771cd-9cb7-4c32-ac68-11c8ffb8f03c" containerName="revoke-keys" Sep 19 10:52:52.140925 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.140743 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="64e771cd-9cb7-4c32-ac68-11c8ffb8f03c" containerName="revoke-keys" Sep 19 10:52:52.140925 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.140800 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a9ac77-d847-4241-b9a7-b935d6855db8" containerName="payload-pre-processing" Sep 19 10:52:52.140925 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.140809 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="99d75e21-0a89-4c94-8cba-030de1f94dbf" containerName="payload-processing" Sep 19 10:52:52.140925 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.140817 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d2dd5b8-8563-4db8-b1e0-6fedc133fbee" containerName="payload-pre-processing" Sep 19 10:52:52.140925 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.140824 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="64e771cd-9cb7-4c32-ac68-11c8ffb8f03c" containerName="revoke-keys" Sep 19 10:52:52.143844 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.143828 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2" Sep 19 10:52:52.146322 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.146301 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 10:52:52.146469 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.146452 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 10:52:52.151110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.151087 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2"] Sep 19 10:52:52.212693 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.212658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65b4b20-23b8-4fd6-8f66-838f42312382-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2\" (UID: \"c65b4b20-23b8-4fd6-8f66-838f42312382\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2" Sep 19 10:52:52.314029 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.313994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65b4b20-23b8-4fd6-8f66-838f42312382-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2\" (UID: \"c65b4b20-23b8-4fd6-8f66-838f42312382\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2" Sep 19 10:52:52.314622 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.314602 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65b4b20-23b8-4fd6-8f66-838f42312382-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2\" (UID: \"c65b4b20-23b8-4fd6-8f66-838f42312382\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2" Sep 19 10:52:52.455863 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.455789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2" Sep 19 10:52:52.725651 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:52.725626 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2"] Sep 19 10:52:52.727884 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:52:52.727857 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65b4b20_23b8_4fd6_8f66_838f42312382.slice/crio-242a3740ee8f2fcbdb9e04571ab714ffad965c18d9a77e4581168a9b78bd9606 WatchSource:0}: Error finding container 242a3740ee8f2fcbdb9e04571ab714ffad965c18d9a77e4581168a9b78bd9606: Status 404 returned error can't find the container with id 242a3740ee8f2fcbdb9e04571ab714ffad965c18d9a77e4581168a9b78bd9606 Sep 19 10:52:53.031089 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.031004 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2" event={"ID":"c65b4b20-23b8-4fd6-8f66-838f42312382","Type":"ContainerStarted","Data":"242a3740ee8f2fcbdb9e04571ab714ffad965c18d9a77e4581168a9b78bd9606"} Sep 19 10:52:53.785562 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.785540 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:52:53.930118 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930076 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnmb8\" (UniqueName: \"kubernetes.io/projected/0215709b-3393-43de-bd07-91da2f6e0bef-kube-api-access-xnmb8\") pod \"0215709b-3393-43de-bd07-91da2f6e0bef\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " Sep 19 10:52:53.930118 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930131 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-istio-data\") pod \"0215709b-3393-43de-bd07-91da2f6e0bef\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " Sep 19 10:52:53.930325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930158 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-workload-certs\") pod \"0215709b-3393-43de-bd07-91da2f6e0bef\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " Sep 19 10:52:53.930325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930192 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0215709b-3393-43de-bd07-91da2f6e0bef-istio-podinfo\") pod \"0215709b-3393-43de-bd07-91da2f6e0bef\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " Sep 19 10:52:53.930325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930233 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-istio-envoy\") pod \"0215709b-3393-43de-bd07-91da2f6e0bef\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " Sep 19 10:52:53.930325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930266 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0215709b-3393-43de-bd07-91da2f6e0bef-istio-token\") pod \"0215709b-3393-43de-bd07-91da2f6e0bef\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " Sep 19 10:52:53.930325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930294 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-credential-socket\") pod \"0215709b-3393-43de-bd07-91da2f6e0bef\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " Sep 19 10:52:53.930325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930319 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0215709b-3393-43de-bd07-91da2f6e0bef-istiod-ca-cert\") pod \"0215709b-3393-43de-bd07-91da2f6e0bef\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " Sep 19 10:52:53.930612 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930339 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-workload-socket\") pod \"0215709b-3393-43de-bd07-91da2f6e0bef\" (UID: \"0215709b-3393-43de-bd07-91da2f6e0bef\") " Sep 19 10:52:53.930612 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930448 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-istio-data" (OuterVolumeSpecName: "istio-data") pod "0215709b-3393-43de-bd07-91da2f6e0bef" (UID: "0215709b-3393-43de-bd07-91da2f6e0bef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:52:53.930612 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930542 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0215709b-3393-43de-bd07-91da2f6e0bef" (UID: "0215709b-3393-43de-bd07-91da2f6e0bef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:52:53.930768 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930695 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:53.930768 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930723 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:53.930896 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930835 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0215709b-3393-43de-bd07-91da2f6e0bef" (UID: "0215709b-3393-43de-bd07-91da2f6e0bef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:52:53.930896 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930880 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0215709b-3393-43de-bd07-91da2f6e0bef" (UID: "0215709b-3393-43de-bd07-91da2f6e0bef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:52:53.931002 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.930919 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0215709b-3393-43de-bd07-91da2f6e0bef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0215709b-3393-43de-bd07-91da2f6e0bef" (UID: "0215709b-3393-43de-bd07-91da2f6e0bef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:52:53.932441 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.932410 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0215709b-3393-43de-bd07-91da2f6e0bef" (UID: "0215709b-3393-43de-bd07-91da2f6e0bef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:52:53.932553 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.932479 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0215709b-3393-43de-bd07-91da2f6e0bef-kube-api-access-xnmb8" (OuterVolumeSpecName: "kube-api-access-xnmb8") pod "0215709b-3393-43de-bd07-91da2f6e0bef" (UID: "0215709b-3393-43de-bd07-91da2f6e0bef"). InnerVolumeSpecName "kube-api-access-xnmb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:52:53.932553 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.932519 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0215709b-3393-43de-bd07-91da2f6e0bef-istio-token" (OuterVolumeSpecName: "istio-token") pod "0215709b-3393-43de-bd07-91da2f6e0bef" (UID: "0215709b-3393-43de-bd07-91da2f6e0bef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:52:53.932631 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:53.932550 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0215709b-3393-43de-bd07-91da2f6e0bef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0215709b-3393-43de-bd07-91da2f6e0bef" (UID: "0215709b-3393-43de-bd07-91da2f6e0bef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 10:52:54.031427 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.031374 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnmb8\" (UniqueName: \"kubernetes.io/projected/0215709b-3393-43de-bd07-91da2f6e0bef-kube-api-access-xnmb8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:54.031427 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.031393 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0215709b-3393-43de-bd07-91da2f6e0bef-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:54.031427 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.031403 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:54.031427 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.031411 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0215709b-3393-43de-bd07-91da2f6e0bef-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:54.031427 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.031419 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:54.031427 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.031426 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0215709b-3393-43de-bd07-91da2f6e0bef-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:54.031896 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.031435 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0215709b-3393-43de-bd07-91da2f6e0bef-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:54.037814 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.037789 2579 generic.go:356] "Generic (PLEG): container finished" podID="c65b4b20-23b8-4fd6-8f66-838f42312382" containerID="dc46d29359936df1cea34b5eaf19453ef50a469be844a2cb92aa137585270672" exitCode=0 Sep 19 10:52:54.037902 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.037880 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2" event={"ID":"c65b4b20-23b8-4fd6-8f66-838f42312382","Type":"ContainerDied","Data":"dc46d29359936df1cea34b5eaf19453ef50a469be844a2cb92aa137585270672"} Sep 19 10:52:54.039635 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.039610 2579 generic.go:356] "Generic (PLEG): container finished" podID="0215709b-3393-43de-bd07-91da2f6e0bef" containerID="ed8c6ed6d617164b49f6794e86f37877ea30b431d94ff050171aad61a54a12dd" exitCode=0 Sep 19 10:52:54.039729 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.039652 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" Sep 19 10:52:54.039729 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.039701 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" event={"ID":"0215709b-3393-43de-bd07-91da2f6e0bef","Type":"ContainerDied","Data":"ed8c6ed6d617164b49f6794e86f37877ea30b431d94ff050171aad61a54a12dd"} Sep 19 10:52:54.039824 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.039770 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k" event={"ID":"0215709b-3393-43de-bd07-91da2f6e0bef","Type":"ContainerDied","Data":"e38f9a1d80ef4119c3ecfbeb5ef709a414c2d6793e165d22c7ffbf857b00fc6a"} Sep 19 10:52:54.039824 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.039800 2579 scope.go:117] "RemoveContainer" containerID="ed8c6ed6d617164b49f6794e86f37877ea30b431d94ff050171aad61a54a12dd" Sep 19 10:52:54.049681 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.049535 2579 scope.go:117] "RemoveContainer" containerID="d770f1dbd0d7679c50362bafb9ee09c666c7c73f0e8f63f12bf8288d4b2f786e" Sep 19 10:52:54.059037 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.059022 2579 scope.go:117] "RemoveContainer" containerID="ed8c6ed6d617164b49f6794e86f37877ea30b431d94ff050171aad61a54a12dd" Sep 19 10:52:54.059398 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:52:54.059378 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8c6ed6d617164b49f6794e86f37877ea30b431d94ff050171aad61a54a12dd\": container with ID starting with ed8c6ed6d617164b49f6794e86f37877ea30b431d94ff050171aad61a54a12dd not found: ID does not exist" containerID="ed8c6ed6d617164b49f6794e86f37877ea30b431d94ff050171aad61a54a12dd" Sep 19 10:52:54.059466 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.059404 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8c6ed6d617164b49f6794e86f37877ea30b431d94ff050171aad61a54a12dd"} err="failed to get container status \"ed8c6ed6d617164b49f6794e86f37877ea30b431d94ff050171aad61a54a12dd\": rpc error: code = NotFound desc = could not find container \"ed8c6ed6d617164b49f6794e86f37877ea30b431d94ff050171aad61a54a12dd\": container with ID starting with ed8c6ed6d617164b49f6794e86f37877ea30b431d94ff050171aad61a54a12dd not found: ID does not exist" Sep 19 10:52:54.059466 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.059421 2579 scope.go:117] "RemoveContainer" containerID="d770f1dbd0d7679c50362bafb9ee09c666c7c73f0e8f63f12bf8288d4b2f786e" Sep 19 10:52:54.059670 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:52:54.059651 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d770f1dbd0d7679c50362bafb9ee09c666c7c73f0e8f63f12bf8288d4b2f786e\": container with ID starting with d770f1dbd0d7679c50362bafb9ee09c666c7c73f0e8f63f12bf8288d4b2f786e not found: ID does not exist" containerID="d770f1dbd0d7679c50362bafb9ee09c666c7c73f0e8f63f12bf8288d4b2f786e" Sep 19 10:52:54.059724 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.059679 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d770f1dbd0d7679c50362bafb9ee09c666c7c73f0e8f63f12bf8288d4b2f786e"} err="failed to get container status \"d770f1dbd0d7679c50362bafb9ee09c666c7c73f0e8f63f12bf8288d4b2f786e\": rpc error: code = NotFound desc = could not find container \"d770f1dbd0d7679c50362bafb9ee09c666c7c73f0e8f63f12bf8288d4b2f786e\": container with ID starting with d770f1dbd0d7679c50362bafb9ee09c666c7c73f0e8f63f12bf8288d4b2f786e not found: ID does not exist" Sep 19 10:52:54.069315 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.067224 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k"] Sep 19 10:52:54.074598 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.074569 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k"] Sep 19 10:52:54.196143 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:54.196115 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" path="/var/lib/kubelet/pods/0215709b-3393-43de-bd07-91da2f6e0bef/volumes" Sep 19 10:52:55.186742 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:55.186719 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2" Sep 19 10:52:55.243767 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:55.243731 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65b4b20-23b8-4fd6-8f66-838f42312382-maas-api-service-ca\") pod \"c65b4b20-23b8-4fd6-8f66-838f42312382\" (UID: \"c65b4b20-23b8-4fd6-8f66-838f42312382\") " Sep 19 10:52:55.244036 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:55.244012 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c65b4b20-23b8-4fd6-8f66-838f42312382-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c65b4b20-23b8-4fd6-8f66-838f42312382" (UID: "c65b4b20-23b8-4fd6-8f66-838f42312382"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:52:55.344617 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:55.344550 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65b4b20-23b8-4fd6-8f66-838f42312382-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:56.053957 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:56.053929 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2" Sep 19 10:52:56.054162 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:56.053969 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2" event={"ID":"c65b4b20-23b8-4fd6-8f66-838f42312382","Type":"ContainerDied","Data":"242a3740ee8f2fcbdb9e04571ab714ffad965c18d9a77e4581168a9b78bd9606"} Sep 19 10:52:56.054162 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:56.054012 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="242a3740ee8f2fcbdb9e04571ab714ffad965c18d9a77e4581168a9b78bd9606" Sep 19 10:52:57.062786 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.062757 2579 generic.go:356] "Generic (PLEG): container finished" podID="4a8f190d-f536-49af-b687-28adddb7733f" containerID="575e52a79f1f70337b626cf28f6869f08a8bc46d6d5b8e4b849822cc274e5ae2" exitCode=0 Sep 19 10:52:57.063136 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.062834 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" event={"ID":"4a8f190d-f536-49af-b687-28adddb7733f","Type":"ContainerDied","Data":"575e52a79f1f70337b626cf28f6869f08a8bc46d6d5b8e4b849822cc274e5ae2"} Sep 19 10:52:57.063136 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.062889 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" event={"ID":"4a8f190d-f536-49af-b687-28adddb7733f","Type":"ContainerDied","Data":"c4d0b675e9163c4882c640e4f569a0f8cc39e07decb2e671ee250b46506f2006"} Sep 19 10:52:57.063136 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.062907 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4d0b675e9163c4882c640e4f569a0f8cc39e07decb2e671ee250b46506f2006" Sep 19 10:52:57.063136 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.062920 2579 scope.go:117] "RemoveContainer" containerID="2d0b4fe430b814274cedbf17ee414369dda787a91b4538446cb9ccec3ee428c2" Sep 19 10:52:57.075713 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.075694 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:52:57.160563 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.160535 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a8f190d-f536-49af-b687-28adddb7733f-istiod-ca-cert\") pod \"4a8f190d-f536-49af-b687-28adddb7733f\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " Sep 19 10:52:57.160737 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.160576 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97rjs\" (UniqueName: \"kubernetes.io/projected/4a8f190d-f536-49af-b687-28adddb7733f-kube-api-access-97rjs\") pod \"4a8f190d-f536-49af-b687-28adddb7733f\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " Sep 19 10:52:57.160737 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.160614 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a8f190d-f536-49af-b687-28adddb7733f-istio-token\") pod \"4a8f190d-f536-49af-b687-28adddb7733f\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " Sep 19 10:52:57.160737 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.160696 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-istio-envoy\") pod \"4a8f190d-f536-49af-b687-28adddb7733f\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " Sep 19 10:52:57.160737 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.160731 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a8f190d-f536-49af-b687-28adddb7733f-istio-podinfo\") pod \"4a8f190d-f536-49af-b687-28adddb7733f\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " Sep 19 10:52:57.160955 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.160759 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-istio-data\") pod \"4a8f190d-f536-49af-b687-28adddb7733f\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " Sep 19 10:52:57.160955 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.160805 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-workload-certs\") pod \"4a8f190d-f536-49af-b687-28adddb7733f\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " Sep 19 10:52:57.160955 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.160838 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-workload-socket\") pod \"4a8f190d-f536-49af-b687-28adddb7733f\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " Sep 19 10:52:57.160955 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.160911 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-credential-socket\") pod \"4a8f190d-f536-49af-b687-28adddb7733f\" (UID: \"4a8f190d-f536-49af-b687-28adddb7733f\") " Sep 19 10:52:57.161173 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.161099 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4a8f190d-f536-49af-b687-28adddb7733f" (UID: "4a8f190d-f536-49af-b687-28adddb7733f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:52:57.161173 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.161116 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-istio-data" (OuterVolumeSpecName: "istio-data") pod "4a8f190d-f536-49af-b687-28adddb7733f" (UID: "4a8f190d-f536-49af-b687-28adddb7733f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:52:57.161349 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.161250 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8f190d-f536-49af-b687-28adddb7733f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4a8f190d-f536-49af-b687-28adddb7733f" (UID: "4a8f190d-f536-49af-b687-28adddb7733f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:52:57.161349 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.161338 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4a8f190d-f536-49af-b687-28adddb7733f" (UID: "4a8f190d-f536-49af-b687-28adddb7733f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:52:57.161447 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.161325 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4a8f190d-f536-49af-b687-28adddb7733f" (UID: "4a8f190d-f536-49af-b687-28adddb7733f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:52:57.161447 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.161365 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:57.161447 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.161395 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:57.162931 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.162904 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8f190d-f536-49af-b687-28adddb7733f-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a8f190d-f536-49af-b687-28adddb7733f" (UID: "4a8f190d-f536-49af-b687-28adddb7733f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:52:57.163053 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.162985 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4a8f190d-f536-49af-b687-28adddb7733f" (UID: "4a8f190d-f536-49af-b687-28adddb7733f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:52:57.163343 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.163318 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a8f190d-f536-49af-b687-28adddb7733f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4a8f190d-f536-49af-b687-28adddb7733f" (UID: "4a8f190d-f536-49af-b687-28adddb7733f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 10:52:57.163343 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.163327 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8f190d-f536-49af-b687-28adddb7733f-kube-api-access-97rjs" (OuterVolumeSpecName: "kube-api-access-97rjs") pod "4a8f190d-f536-49af-b687-28adddb7733f" (UID: "4a8f190d-f536-49af-b687-28adddb7733f"). InnerVolumeSpecName "kube-api-access-97rjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:52:57.262976 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.262928 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:57.262976 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.262957 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:57.262976 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.262971 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a8f190d-f536-49af-b687-28adddb7733f-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:57.262976 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.262985 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97rjs\" (UniqueName: \"kubernetes.io/projected/4a8f190d-f536-49af-b687-28adddb7733f-kube-api-access-97rjs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:57.263265 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.262998 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a8f190d-f536-49af-b687-28adddb7733f-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:57.263265 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.263011 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a8f190d-f536-49af-b687-28adddb7733f-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:57.263265 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:57.263023 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a8f190d-f536-49af-b687-28adddb7733f-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:52:58.069453 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:58.069422 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8" Sep 19 10:52:58.100586 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:58.100555 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8"] Sep 19 10:52:58.103456 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:58.103432 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8"] Sep 19 10:52:58.196381 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:52:58.196343 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a8f190d-f536-49af-b687-28adddb7733f" path="/var/lib/kubelet/pods/4a8f190d-f536-49af-b687-28adddb7733f/volumes" Sep 19 10:53:03.030787 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.030743 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2"] Sep 19 10:53:03.032892 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.032869 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6cf6a0-94cs2"] Sep 19 10:53:03.142036 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.141996 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d"] Sep 19 10:53:03.142704 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142679 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" containerName="istio-proxy" Sep 19 10:53:03.142704 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142705 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" containerName="istio-proxy" Sep 19 10:53:03.142842 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142729 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" Sep 19 10:53:03.142842 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142738 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" Sep 19 10:53:03.142842 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142750 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" Sep 19 10:53:03.142842 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142760 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" Sep 19 10:53:03.142842 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142782 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" containerName="istio-proxy" Sep 19 10:53:03.142842 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142790 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" containerName="istio-proxy" Sep 19 10:53:03.142842 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142800 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c65b4b20-23b8-4fd6-8f66-838f42312382" containerName="revoke-keys" Sep 19 10:53:03.142842 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142809 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c65b4b20-23b8-4fd6-8f66-838f42312382" containerName="revoke-keys" Sep 19 10:53:03.143146 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142904 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" containerName="istio-proxy" Sep 19 10:53:03.143146 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142917 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0215709b-3393-43de-bd07-91da2f6e0bef" containerName="istio-proxy" Sep 19 10:53:03.143146 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142927 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" Sep 19 10:53:03.143146 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142935 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8f190d-f536-49af-b687-28adddb7733f" containerName="istio-proxy" Sep 19 10:53:03.143146 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.142944 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c65b4b20-23b8-4fd6-8f66-838f42312382" containerName="revoke-keys" Sep 19 10:53:03.147751 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.147728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.150366 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.150346 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-9ac6b3-openshift-default-dockercfg-sg2f7\"" Sep 19 10:53:03.156641 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.156618 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d"] Sep 19 10:53:03.320446 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.320346 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-token\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.320446 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.320405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-data\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.320652 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.320497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkvgk\" (UniqueName: \"kubernetes.io/projected/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-kube-api-access-jkvgk\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.320652 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.320565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-workload-certs\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.320652 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.320599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-envoy\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.320652 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.320618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istiod-ca-cert\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.320652 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.320641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-workload-socket\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.320881 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.320675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-credential-socket\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.320881 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.320727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-podinfo\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.421829 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.421796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-workload-certs\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.422023 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.421865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-envoy\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.422023 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.421897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istiod-ca-cert\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.422023 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.421930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-workload-socket\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.422023 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.421949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-credential-socket\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.422023 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.421986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-podinfo\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.422350 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.422046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-token\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.422350 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.422114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-data\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.422350 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.422185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkvgk\" (UniqueName: \"kubernetes.io/projected/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-kube-api-access-jkvgk\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.422350 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.422328 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-workload-certs\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.422642 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.422601 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-data\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.422725 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.422690 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-credential-socket\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.422784 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.422746 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istiod-ca-cert\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.422868 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.422847 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-workload-socket\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.424404 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.424384 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-envoy\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.424654 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.424635 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-podinfo\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.429312 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.429293 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-token\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.429491 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.429476 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkvgk\" (UniqueName: \"kubernetes.io/projected/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-kube-api-access-jkvgk\") pod \"e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.462649 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.462614 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:03.747248 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.747222 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d"] Sep 19 10:53:03.749044 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:53:03.749019 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351bfa27_6ecf_4ad7_98d9_2d2a1a7027a8.slice/crio-b9244664519dff300d2f5fde40aca72d1bbea9f7508b327ef4cf800e460ac02a WatchSource:0}: Error finding container b9244664519dff300d2f5fde40aca72d1bbea9f7508b327ef4cf800e460ac02a: Status 404 returned error can't find the container with id b9244664519dff300d2f5fde40aca72d1bbea9f7508b327ef4cf800e460ac02a Sep 19 10:53:03.751439 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.751407 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:03.751529 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.751479 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:03.751529 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:03.751509 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:04.105585 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:04.105500 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" event={"ID":"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8","Type":"ContainerStarted","Data":"f5a4cb03009f987c99a4691b0b02623b513d3e492049a8667bb359d39f6e3018"} Sep 19 10:53:04.105585 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:04.105557 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" event={"ID":"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8","Type":"ContainerStarted","Data":"b9244664519dff300d2f5fde40aca72d1bbea9f7508b327ef4cf800e460ac02a"} Sep 19 10:53:04.125157 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:04.125107 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" podStartSLOduration=1.125092524 podStartE2EDuration="1.125092524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:53:04.122865574 +0000 UTC m=+1116.035370292" watchObservedRunningTime="2026-09-19 10:53:04.125092524 +0000 UTC m=+1116.037597240" Sep 19 10:53:04.195807 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:04.195770 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c65b4b20-23b8-4fd6-8f66-838f42312382" path="/var/lib/kubelet/pods/c65b4b20-23b8-4fd6-8f66-838f42312382/volumes" Sep 19 10:53:04.463129 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:04.463096 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:04.468020 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:04.467997 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:05.111692 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:05.111663 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:05.112629 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:05.112610 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:09.471087 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:09.471017 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr"] Sep 19 10:53:09.471534 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:09.471299 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" podUID="ddd05d1e-9759-421e-826f-44441269f943" containerName="istio-proxy" containerID="cri-o://7bea3617020b32c075cfc2dca898cdf5ee4e492b766d330f926a832e03ceb351" gracePeriod=30 Sep 19 10:53:10.265011 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.264977 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt"] Sep 19 10:53:10.269986 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.269958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.272623 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.272596 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-72e262-openshift-default-dockercfg-8nzdv\"" Sep 19 10:53:10.280671 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.280646 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt"] Sep 19 10:53:10.290178 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.290153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-envoy\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.290310 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.290195 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-workload-certs\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.290310 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.290233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-podinfo\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.290310 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.290264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-workload-socket\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.290310 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.290279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-credential-socket\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.290559 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.290356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5096ee-2458-4ac4-9f7c-701756174f18-istiod-ca-cert\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.290559 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.290412 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-token\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.290559 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.290467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-data\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.290559 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.290544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pvjk\" (UniqueName: \"kubernetes.io/projected/2b5096ee-2458-4ac4-9f7c-701756174f18-kube-api-access-4pvjk\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.391168 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.391134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-envoy\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.391371 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.391188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-workload-certs\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.391371 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.391216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-podinfo\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.391371 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.391243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-workload-socket\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.391371 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.391259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-credential-socket\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.391371 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.391295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5096ee-2458-4ac4-9f7c-701756174f18-istiod-ca-cert\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.391371 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.391325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-token\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.391699 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.391596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-data\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.391699 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.391666 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-workload-certs\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.391699 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.391674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pvjk\" (UniqueName: \"kubernetes.io/projected/2b5096ee-2458-4ac4-9f7c-701756174f18-kube-api-access-4pvjk\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.391846 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.391709 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-workload-socket\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.392008 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.391983 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-credential-socket\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.392219 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.392053 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-data\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.392219 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.392122 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5096ee-2458-4ac4-9f7c-701756174f18-istiod-ca-cert\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.393712 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.393682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-envoy\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.394026 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.394006 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-podinfo\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.411315 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.411294 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-token\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.411536 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.411519 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pvjk\" (UniqueName: \"kubernetes.io/projected/2b5096ee-2458-4ac4-9f7c-701756174f18-kube-api-access-4pvjk\") pod \"e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.582269 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.582182 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:10.863838 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.863810 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt"] Sep 19 10:53:10.865909 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:53:10.865884 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5096ee_2458_4ac4_9f7c_701756174f18.slice/crio-2d0ed5f61cc92f66cac6f5c830e6701a70b22b14f43e418081367c3780d9a79e WatchSource:0}: Error finding container 2d0ed5f61cc92f66cac6f5c830e6701a70b22b14f43e418081367c3780d9a79e: Status 404 returned error can't find the container with id 2d0ed5f61cc92f66cac6f5c830e6701a70b22b14f43e418081367c3780d9a79e Sep 19 10:53:10.868121 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.868085 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:10.868204 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.868157 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:10.868204 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:10.868187 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:11.153650 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:11.153611 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" event={"ID":"2b5096ee-2458-4ac4-9f7c-701756174f18","Type":"ContainerStarted","Data":"806fae03de6b58ae179776ea51843ad6b0ab62c54bb3aeb8ee21f52e6b786906"} Sep 19 10:53:11.153849 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:11.153680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" event={"ID":"2b5096ee-2458-4ac4-9f7c-701756174f18","Type":"ContainerStarted","Data":"2d0ed5f61cc92f66cac6f5c830e6701a70b22b14f43e418081367c3780d9a79e"} Sep 19 10:53:11.173789 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:11.173737 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" podStartSLOduration=1.173716511 podStartE2EDuration="1.173716511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:53:11.172529217 +0000 UTC m=+1123.085033959" watchObservedRunningTime="2026-09-19 10:53:11.173716511 +0000 UTC m=+1123.086221228" Sep 19 10:53:11.583040 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:11.582957 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:11.588467 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:11.588441 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:12.159577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:12.159538 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:12.160516 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:12.160494 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:14.724676 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.724653 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:53:14.732540 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.732519 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-credential-socket\") pod \"ddd05d1e-9759-421e-826f-44441269f943\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " Sep 19 10:53:14.732666 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.732557 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-workload-certs\") pod \"ddd05d1e-9759-421e-826f-44441269f943\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " Sep 19 10:53:14.732666 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.732596 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd05d1e-9759-421e-826f-44441269f943-istiod-ca-cert\") pod \"ddd05d1e-9759-421e-826f-44441269f943\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " Sep 19 10:53:14.732666 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.732622 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd05d1e-9759-421e-826f-44441269f943-istio-token\") pod \"ddd05d1e-9759-421e-826f-44441269f943\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " Sep 19 10:53:14.732830 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.732686 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-workload-socket\") pod \"ddd05d1e-9759-421e-826f-44441269f943\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " Sep 19 10:53:14.732830 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.732719 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-istio-data\") pod \"ddd05d1e-9759-421e-826f-44441269f943\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " Sep 19 10:53:14.732830 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.732792 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd05d1e-9759-421e-826f-44441269f943-istio-podinfo\") pod \"ddd05d1e-9759-421e-826f-44441269f943\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " Sep 19 10:53:14.732830 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.732819 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ddd05d1e-9759-421e-826f-44441269f943" (UID: "ddd05d1e-9759-421e-826f-44441269f943"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:53:14.733051 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.732832 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ddd05d1e-9759-421e-826f-44441269f943" (UID: "ddd05d1e-9759-421e-826f-44441269f943"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:53:14.733051 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.732849 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w9p8\" (UniqueName: \"kubernetes.io/projected/ddd05d1e-9759-421e-826f-44441269f943-kube-api-access-5w9p8\") pod \"ddd05d1e-9759-421e-826f-44441269f943\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " Sep 19 10:53:14.733051 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.732877 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-istio-envoy\") pod \"ddd05d1e-9759-421e-826f-44441269f943\" (UID: \"ddd05d1e-9759-421e-826f-44441269f943\") " Sep 19 10:53:14.733051 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.732975 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd05d1e-9759-421e-826f-44441269f943-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ddd05d1e-9759-421e-826f-44441269f943" (UID: "ddd05d1e-9759-421e-826f-44441269f943"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:53:14.733051 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.733004 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-istio-data" (OuterVolumeSpecName: "istio-data") pod "ddd05d1e-9759-421e-826f-44441269f943" (UID: "ddd05d1e-9759-421e-826f-44441269f943"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:53:14.733347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.733109 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ddd05d1e-9759-421e-826f-44441269f943" (UID: "ddd05d1e-9759-421e-826f-44441269f943"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:53:14.733347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.733208 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:53:14.733347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.733223 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:53:14.733347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.733239 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:53:14.733347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.733258 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:53:14.733347 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.733275 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd05d1e-9759-421e-826f-44441269f943-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:53:14.734954 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.734916 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd05d1e-9759-421e-826f-44441269f943-istio-token" (OuterVolumeSpecName: "istio-token") pod "ddd05d1e-9759-421e-826f-44441269f943" (UID: "ddd05d1e-9759-421e-826f-44441269f943"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:53:14.735141 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.734975 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ddd05d1e-9759-421e-826f-44441269f943" (UID: "ddd05d1e-9759-421e-826f-44441269f943"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:53:14.735141 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.734985 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddd05d1e-9759-421e-826f-44441269f943-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ddd05d1e-9759-421e-826f-44441269f943" (UID: "ddd05d1e-9759-421e-826f-44441269f943"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 10:53:14.735500 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.735474 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd05d1e-9759-421e-826f-44441269f943-kube-api-access-5w9p8" (OuterVolumeSpecName: "kube-api-access-5w9p8") pod "ddd05d1e-9759-421e-826f-44441269f943" (UID: "ddd05d1e-9759-421e-826f-44441269f943"). InnerVolumeSpecName "kube-api-access-5w9p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:53:14.833951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.833918 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd05d1e-9759-421e-826f-44441269f943-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:53:14.833951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.833944 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd05d1e-9759-421e-826f-44441269f943-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:53:14.833951 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.833954 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w9p8\" (UniqueName: \"kubernetes.io/projected/ddd05d1e-9759-421e-826f-44441269f943-kube-api-access-5w9p8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:53:14.834182 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:14.833962 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd05d1e-9759-421e-826f-44441269f943-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:53:15.178816 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:15.178783 2579 generic.go:356] "Generic (PLEG): container finished" podID="ddd05d1e-9759-421e-826f-44441269f943" containerID="7bea3617020b32c075cfc2dca898cdf5ee4e492b766d330f926a832e03ceb351" exitCode=0 Sep 19 10:53:15.178992 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:15.178830 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" Sep 19 10:53:15.178992 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:15.178864 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" event={"ID":"ddd05d1e-9759-421e-826f-44441269f943","Type":"ContainerDied","Data":"7bea3617020b32c075cfc2dca898cdf5ee4e492b766d330f926a832e03ceb351"} Sep 19 10:53:15.178992 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:15.178917 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr" event={"ID":"ddd05d1e-9759-421e-826f-44441269f943","Type":"ContainerDied","Data":"170c35f98987a8f9e2b8e4e5227a0bd08fc90b3209b607e8eca38e7e9cd8a744"} Sep 19 10:53:15.178992 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:15.178938 2579 scope.go:117] "RemoveContainer" containerID="7bea3617020b32c075cfc2dca898cdf5ee4e492b766d330f926a832e03ceb351" Sep 19 10:53:15.189245 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:15.189227 2579 scope.go:117] "RemoveContainer" containerID="ab252c1636a12eda2a494db82bf5f31ad7390766aad67927a563908b73d60e9b" Sep 19 10:53:15.200694 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:15.200674 2579 scope.go:117] "RemoveContainer" containerID="7bea3617020b32c075cfc2dca898cdf5ee4e492b766d330f926a832e03ceb351" Sep 19 10:53:15.200963 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:53:15.200944 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bea3617020b32c075cfc2dca898cdf5ee4e492b766d330f926a832e03ceb351\": container with ID starting with 7bea3617020b32c075cfc2dca898cdf5ee4e492b766d330f926a832e03ceb351 not found: ID does not exist" containerID="7bea3617020b32c075cfc2dca898cdf5ee4e492b766d330f926a832e03ceb351" Sep 19 10:53:15.201012 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:15.200974 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bea3617020b32c075cfc2dca898cdf5ee4e492b766d330f926a832e03ceb351"} err="failed to get container status \"7bea3617020b32c075cfc2dca898cdf5ee4e492b766d330f926a832e03ceb351\": rpc error: code = NotFound desc = could not find container \"7bea3617020b32c075cfc2dca898cdf5ee4e492b766d330f926a832e03ceb351\": container with ID starting with 7bea3617020b32c075cfc2dca898cdf5ee4e492b766d330f926a832e03ceb351 not found: ID does not exist" Sep 19 10:53:15.201012 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:15.200990 2579 scope.go:117] "RemoveContainer" containerID="ab252c1636a12eda2a494db82bf5f31ad7390766aad67927a563908b73d60e9b" Sep 19 10:53:15.201160 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:15.201139 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr"] Sep 19 10:53:15.201268 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:53:15.201245 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab252c1636a12eda2a494db82bf5f31ad7390766aad67927a563908b73d60e9b\": container with ID starting with ab252c1636a12eda2a494db82bf5f31ad7390766aad67927a563908b73d60e9b not found: ID does not exist" containerID="ab252c1636a12eda2a494db82bf5f31ad7390766aad67927a563908b73d60e9b" Sep 19 10:53:15.201318 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:15.201266 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab252c1636a12eda2a494db82bf5f31ad7390766aad67927a563908b73d60e9b"} err="failed to get container status \"ab252c1636a12eda2a494db82bf5f31ad7390766aad67927a563908b73d60e9b\": rpc error: code = NotFound desc = could not find container \"ab252c1636a12eda2a494db82bf5f31ad7390766aad67927a563908b73d60e9b\": container with ID starting with ab252c1636a12eda2a494db82bf5f31ad7390766aad67927a563908b73d60e9b not found: ID does not exist" Sep 19 10:53:15.203051 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:15.203033 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr"] Sep 19 10:53:16.196436 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.196392 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd05d1e-9759-421e-826f-44441269f943" path="/var/lib/kubelet/pods/ddd05d1e-9759-421e-826f-44441269f943/volumes" Sep 19 10:53:16.487998 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.487914 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28"] Sep 19 10:53:16.488351 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.488338 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd05d1e-9759-421e-826f-44441269f943" containerName="istio-proxy" Sep 19 10:53:16.488396 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.488354 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd05d1e-9759-421e-826f-44441269f943" containerName="istio-proxy" Sep 19 10:53:16.488451 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.488441 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd05d1e-9759-421e-826f-44441269f943" containerName="istio-proxy" Sep 19 10:53:16.488491 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.488455 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd05d1e-9759-421e-826f-44441269f943" containerName="istio-proxy" Sep 19 10:53:16.493015 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.492996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" Sep 19 10:53:16.497992 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.497972 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-72e262\"" Sep 19 10:53:16.498089 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.498039 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-72e262-dockercfg-4nlds\"" Sep 19 10:53:16.504759 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.504730 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28"] Sep 19 10:53:16.522200 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.522175 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb"] Sep 19 10:53:16.522818 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.522799 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd05d1e-9759-421e-826f-44441269f943" containerName="istio-proxy" Sep 19 10:53:16.522864 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.522823 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd05d1e-9759-421e-826f-44441269f943" containerName="istio-proxy" Sep 19 10:53:16.526474 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.526453 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" Sep 19 10:53:16.533086 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.533048 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb"] Sep 19 10:53:16.551348 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.551318 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g7tp\" (UniqueName: \"kubernetes.io/projected/46ede73c-f5b1-4625-8073-fc5401026a40-kube-api-access-8g7tp\") pod \"payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb\" (UID: \"46ede73c-f5b1-4625-8073-fc5401026a40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" Sep 19 10:53:16.551478 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.551398 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9af77da-8c73-477a-9db8-1f12e966f951-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28\" (UID: \"b9af77da-8c73-477a-9db8-1f12e966f951\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" Sep 19 10:53:16.551478 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.551433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j4mm\" (UniqueName: \"kubernetes.io/projected/b9af77da-8c73-477a-9db8-1f12e966f951-kube-api-access-2j4mm\") pod \"payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28\" (UID: \"b9af77da-8c73-477a-9db8-1f12e966f951\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" Sep 19 10:53:16.551478 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.551469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ede73c-f5b1-4625-8073-fc5401026a40-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb\" (UID: \"46ede73c-f5b1-4625-8073-fc5401026a40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" Sep 19 10:53:16.652975 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.652942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9af77da-8c73-477a-9db8-1f12e966f951-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28\" (UID: \"b9af77da-8c73-477a-9db8-1f12e966f951\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" Sep 19 10:53:16.653197 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.653005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j4mm\" (UniqueName: \"kubernetes.io/projected/b9af77da-8c73-477a-9db8-1f12e966f951-kube-api-access-2j4mm\") pod \"payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28\" (UID: \"b9af77da-8c73-477a-9db8-1f12e966f951\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" Sep 19 10:53:16.653197 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.653076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ede73c-f5b1-4625-8073-fc5401026a40-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb\" (UID: \"46ede73c-f5b1-4625-8073-fc5401026a40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" Sep 19 10:53:16.653197 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.653160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g7tp\" (UniqueName: \"kubernetes.io/projected/46ede73c-f5b1-4625-8073-fc5401026a40-kube-api-access-8g7tp\") pod \"payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb\" (UID: \"46ede73c-f5b1-4625-8073-fc5401026a40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" Sep 19 10:53:16.653837 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.653810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9af77da-8c73-477a-9db8-1f12e966f951-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28\" (UID: \"b9af77da-8c73-477a-9db8-1f12e966f951\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" Sep 19 10:53:16.653986 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.653965 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ede73c-f5b1-4625-8073-fc5401026a40-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb\" (UID: \"46ede73c-f5b1-4625-8073-fc5401026a40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" Sep 19 10:53:16.660920 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.660888 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j4mm\" (UniqueName: \"kubernetes.io/projected/b9af77da-8c73-477a-9db8-1f12e966f951-kube-api-access-2j4mm\") pod \"payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28\" (UID: \"b9af77da-8c73-477a-9db8-1f12e966f951\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" Sep 19 10:53:16.661036 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.660898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g7tp\" (UniqueName: \"kubernetes.io/projected/46ede73c-f5b1-4625-8073-fc5401026a40-kube-api-access-8g7tp\") pod \"payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb\" (UID: \"46ede73c-f5b1-4625-8073-fc5401026a40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" Sep 19 10:53:16.804892 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.804809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" Sep 19 10:53:16.841463 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:16.841210 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" Sep 19 10:53:17.095003 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:17.094974 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28"] Sep 19 10:53:17.097699 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:53:17.097672 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9af77da_8c73_477a_9db8_1f12e966f951.slice/crio-2b194800d946ba0b26ed4222d267692628933ca064cad65264f97e8014270ce7 WatchSource:0}: Error finding container 2b194800d946ba0b26ed4222d267692628933ca064cad65264f97e8014270ce7: Status 404 returned error can't find the container with id 2b194800d946ba0b26ed4222d267692628933ca064cad65264f97e8014270ce7 Sep 19 10:53:17.143715 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:17.143684 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb"] Sep 19 10:53:17.144946 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:53:17.144915 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ede73c_f5b1_4625_8073_fc5401026a40.slice/crio-6f43ffc7f4268e2ef444e32245b21b0d3b0157a7449a1d7c6b7e1fca7bc32d60 WatchSource:0}: Error finding container 6f43ffc7f4268e2ef444e32245b21b0d3b0157a7449a1d7c6b7e1fca7bc32d60: Status 404 returned error can't find the container with id 6f43ffc7f4268e2ef444e32245b21b0d3b0157a7449a1d7c6b7e1fca7bc32d60 Sep 19 10:53:17.196002 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:17.195973 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" event={"ID":"46ede73c-f5b1-4625-8073-fc5401026a40","Type":"ContainerStarted","Data":"6f43ffc7f4268e2ef444e32245b21b0d3b0157a7449a1d7c6b7e1fca7bc32d60"} Sep 19 10:53:17.197503 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:17.197482 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" event={"ID":"b9af77da-8c73-477a-9db8-1f12e966f951","Type":"ContainerStarted","Data":"a08c561eda8273dbc9a30f074f6b809a76f914ec81987b2e1193745a36ad349c"} Sep 19 10:53:17.197792 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:17.197519 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" event={"ID":"b9af77da-8c73-477a-9db8-1f12e966f951","Type":"ContainerStarted","Data":"2b194800d946ba0b26ed4222d267692628933ca064cad65264f97e8014270ce7"} Sep 19 10:53:17.197792 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:17.197598 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" Sep 19 10:53:17.215894 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:17.215850 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" podStartSLOduration=1.21583504 podStartE2EDuration="1.21583504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:53:17.213669484 +0000 UTC m=+1129.126174202" watchObservedRunningTime="2026-09-19 10:53:17.21583504 +0000 UTC m=+1129.128339781" Sep 19 10:53:18.205021 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:18.204984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" event={"ID":"46ede73c-f5b1-4625-8073-fc5401026a40","Type":"ContainerStarted","Data":"071de16aa68c6ac8ca62d4f93c28e24639683d434290acb361f7d0df07b0313e"} Sep 19 10:53:18.205470 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:18.205310 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" Sep 19 10:53:18.221192 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:18.221149 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" podStartSLOduration=2.221134482 podStartE2EDuration="2.221134482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:53:18.219488321 +0000 UTC m=+1130.131993047" watchObservedRunningTime="2026-09-19 10:53:18.221134482 +0000 UTC m=+1130.133639198" Sep 19 10:53:26.255553 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:26.255521 2579 generic.go:356] "Generic (PLEG): container finished" podID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerID="f5a4cb03009f987c99a4691b0b02623b513d3e492049a8667bb359d39f6e3018" exitCode=0 Sep 19 10:53:26.255933 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:26.255592 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" event={"ID":"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8","Type":"ContainerDied","Data":"f5a4cb03009f987c99a4691b0b02623b513d3e492049a8667bb359d39f6e3018"} Sep 19 10:53:26.255982 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:26.255972 2579 scope.go:117] "RemoveContainer" containerID="f5a4cb03009f987c99a4691b0b02623b513d3e492049a8667bb359d39f6e3018" Sep 19 10:53:26.257156 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:26.257130 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:26.257225 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:26.257193 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:26.257263 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:26.257224 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:27.267807 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:27.267760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" event={"ID":"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8","Type":"ContainerStarted","Data":"e3cb1288f5e9279764d059a9479f56c808bf39e8560a4803782833814e823ea7"} Sep 19 10:53:27.463039 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:27.462999 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:27.463325 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:27.463297 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 19 10:53:27.463407 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:27.463367 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" podUID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 19 10:53:28.207228 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:28.207202 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" Sep 19 10:53:28.464136 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:28.463532 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 19 10:53:28.464136 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:28.463588 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" podUID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 19 10:53:29.214038 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:29.214003 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" Sep 19 10:53:29.463220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:29.463179 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 19 10:53:29.463398 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:29.463249 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" podUID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 19 10:53:30.463478 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:30.463437 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 19 10:53:30.463956 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:30.463512 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" podUID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 19 10:53:31.464033 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.463984 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 19 10:53:31.464556 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.464086 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" podUID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 19 10:53:31.579804 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.579768 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j"] Sep 19 10:53:31.591878 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.585787 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j"] Sep 19 10:53:31.591878 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.585909 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.591878 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.591124 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-266c89-openshift-default-dockercfg-zn62j\"" Sep 19 10:53:31.712502 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.712461 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-workload-socket\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.712701 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.712537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-credential-socket\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.712701 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.712572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-workload-certs\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.712701 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.712592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-data\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.712701 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.712614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-podinfo\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.712701 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.712668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istiod-ca-cert\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.712927 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.712727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-token\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.712927 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.712772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-envoy\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.712927 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.712797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5gw4\" (UniqueName: \"kubernetes.io/projected/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-kube-api-access-t5gw4\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.813708 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.813607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istiod-ca-cert\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.813708 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.813652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-token\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.813708 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.813678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-envoy\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.814007 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.813707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5gw4\" (UniqueName: \"kubernetes.io/projected/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-kube-api-access-t5gw4\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.814007 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.813785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-workload-socket\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.814007 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.813847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-credential-socket\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.814007 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.813882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-workload-certs\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.814007 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.813908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-data\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.814007 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.813935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-podinfo\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.814350 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.814191 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-workload-socket\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.814350 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.814236 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-credential-socket\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.814464 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.814433 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istiod-ca-cert\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.814526 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.814495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-workload-certs\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.814587 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.814529 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-data\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.816328 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.816300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-envoy\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.816562 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.816543 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-podinfo\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.823596 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.823574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5gw4\" (UniqueName: \"kubernetes.io/projected/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-kube-api-access-t5gw4\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.823716 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.823635 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-token\") pod \"e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:31.918467 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:31.918200 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:32.223916 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:32.223889 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j"] Sep 19 10:53:32.225424 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:53:32.225395 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c33bf4e_8864_4014_90e5_cb5cf43a07b6.slice/crio-7a583e101b3ccdead066421ca052d5fe469547bdd71c33e94fa537971ae1bc7e WatchSource:0}: Error finding container 7a583e101b3ccdead066421ca052d5fe469547bdd71c33e94fa537971ae1bc7e: Status 404 returned error can't find the container with id 7a583e101b3ccdead066421ca052d5fe469547bdd71c33e94fa537971ae1bc7e Sep 19 10:53:32.227704 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:32.227674 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:32.227769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:32.227756 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:32.227809 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:32.227786 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:32.300854 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:32.300822 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" event={"ID":"0c33bf4e-8864-4014-90e5-cb5cf43a07b6","Type":"ContainerStarted","Data":"7a583e101b3ccdead066421ca052d5fe469547bdd71c33e94fa537971ae1bc7e"} Sep 19 10:53:32.605250 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:32.605167 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:32.605704 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:32.605621 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:32.606739 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:32.606714 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 10:53:33.308380 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:33.308346 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" event={"ID":"0c33bf4e-8864-4014-90e5-cb5cf43a07b6","Type":"ContainerStarted","Data":"d87a47f6a811a0dc5f883be378a6c8ecb66974ae8ec23a5a1d4b93260b37309c"} Sep 19 10:53:33.328622 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:33.328577 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" podStartSLOduration=2.328563466 podStartE2EDuration="2.328563466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:53:33.326624805 +0000 UTC m=+1145.239129522" watchObservedRunningTime="2026-09-19 10:53:33.328563466 +0000 UTC m=+1145.241068182" Sep 19 10:53:33.919931 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:33.919894 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:33.925194 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:33.925170 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:34.314163 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:34.314049 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:34.314983 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:34.314965 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:53:38.803557 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:38.803526 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl"] Sep 19 10:53:38.808293 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:38.808271 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" Sep 19 10:53:38.810830 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:38.810805 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-266c89-dockercfg-jf5b6\"" Sep 19 10:53:38.810924 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:38.810808 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-266c89\"" Sep 19 10:53:38.816423 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:38.816395 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl"] Sep 19 10:53:38.892569 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:38.892530 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24815e6-1b76-4f80-80bb-f3b5c25c866a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl\" (UID: \"c24815e6-1b76-4f80-80bb-f3b5c25c866a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" Sep 19 10:53:38.892883 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:38.892854 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcrvn\" (UniqueName: \"kubernetes.io/projected/c24815e6-1b76-4f80-80bb-f3b5c25c866a-kube-api-access-dcrvn\") pod \"payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl\" (UID: \"c24815e6-1b76-4f80-80bb-f3b5c25c866a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" Sep 19 10:53:38.993934 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:38.993899 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24815e6-1b76-4f80-80bb-f3b5c25c866a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl\" (UID: \"c24815e6-1b76-4f80-80bb-f3b5c25c866a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" Sep 19 10:53:38.994132 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:38.993977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcrvn\" (UniqueName: \"kubernetes.io/projected/c24815e6-1b76-4f80-80bb-f3b5c25c866a-kube-api-access-dcrvn\") pod \"payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl\" (UID: \"c24815e6-1b76-4f80-80bb-f3b5c25c866a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" Sep 19 10:53:38.994655 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:38.994636 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24815e6-1b76-4f80-80bb-f3b5c25c866a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl\" (UID: \"c24815e6-1b76-4f80-80bb-f3b5c25c866a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" Sep 19 10:53:39.001643 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:39.001624 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcrvn\" (UniqueName: \"kubernetes.io/projected/c24815e6-1b76-4f80-80bb-f3b5c25c866a-kube-api-access-dcrvn\") pod \"payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl\" (UID: \"c24815e6-1b76-4f80-80bb-f3b5c25c866a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" Sep 19 10:53:39.120636 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:39.120548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" Sep 19 10:53:39.417126 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:39.417102 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl"] Sep 19 10:53:39.418985 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:53:39.418956 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24815e6_1b76_4f80_80bb_f3b5c25c866a.slice/crio-5c27649c6c4fa9210776a0ef97938f1e30dd61ff4833ada1e11a4a2ec8ad5de3 WatchSource:0}: Error finding container 5c27649c6c4fa9210776a0ef97938f1e30dd61ff4833ada1e11a4a2ec8ad5de3: Status 404 returned error can't find the container with id 5c27649c6c4fa9210776a0ef97938f1e30dd61ff4833ada1e11a4a2ec8ad5de3 Sep 19 10:53:39.924189 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:39.924152 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb"] Sep 19 10:53:39.927802 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:39.927783 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" Sep 19 10:53:39.930653 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:39.930631 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-99b43e\"" Sep 19 10:53:39.930757 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:39.930632 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-99b43e-dockercfg-ggwsl\"" Sep 19 10:53:39.938819 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:39.938797 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb"] Sep 19 10:53:39.949242 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:39.949220 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh"] Sep 19 10:53:39.981892 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:39.981864 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" Sep 19 10:53:39.983019 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:39.982995 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh"] Sep 19 10:53:40.005665 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.005643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06314f8b-a1f2-44f9-ab95-52cc7919e6fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb\" (UID: \"06314f8b-a1f2-44f9-ab95-52cc7919e6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" Sep 19 10:53:40.005771 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.005696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x96rw\" (UniqueName: \"kubernetes.io/projected/06314f8b-a1f2-44f9-ab95-52cc7919e6fd-kube-api-access-x96rw\") pod \"payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb\" (UID: \"06314f8b-a1f2-44f9-ab95-52cc7919e6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" Sep 19 10:53:40.107181 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.107143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba8bb4c-a739-480c-9ae5-8d3213c86abc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh\" (UID: \"aba8bb4c-a739-480c-9ae5-8d3213c86abc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" Sep 19 10:53:40.107356 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.107188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x96rw\" (UniqueName: \"kubernetes.io/projected/06314f8b-a1f2-44f9-ab95-52cc7919e6fd-kube-api-access-x96rw\") pod \"payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb\" (UID: \"06314f8b-a1f2-44f9-ab95-52cc7919e6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" Sep 19 10:53:40.107400 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.107377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9jz4\" (UniqueName: \"kubernetes.io/projected/aba8bb4c-a739-480c-9ae5-8d3213c86abc-kube-api-access-p9jz4\") pod \"payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh\" (UID: \"aba8bb4c-a739-480c-9ae5-8d3213c86abc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" Sep 19 10:53:40.107458 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.107441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06314f8b-a1f2-44f9-ab95-52cc7919e6fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb\" (UID: \"06314f8b-a1f2-44f9-ab95-52cc7919e6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" Sep 19 10:53:40.108099 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.108082 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06314f8b-a1f2-44f9-ab95-52cc7919e6fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb\" (UID: \"06314f8b-a1f2-44f9-ab95-52cc7919e6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" Sep 19 10:53:40.115048 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.115022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x96rw\" (UniqueName: \"kubernetes.io/projected/06314f8b-a1f2-44f9-ab95-52cc7919e6fd-kube-api-access-x96rw\") pod \"payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb\" (UID: \"06314f8b-a1f2-44f9-ab95-52cc7919e6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" Sep 19 10:53:40.208118 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.208027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9jz4\" (UniqueName: \"kubernetes.io/projected/aba8bb4c-a739-480c-9ae5-8d3213c86abc-kube-api-access-p9jz4\") pod \"payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh\" (UID: \"aba8bb4c-a739-480c-9ae5-8d3213c86abc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" Sep 19 10:53:40.208256 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.208123 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba8bb4c-a739-480c-9ae5-8d3213c86abc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh\" (UID: \"aba8bb4c-a739-480c-9ae5-8d3213c86abc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" Sep 19 10:53:40.208769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.208748 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba8bb4c-a739-480c-9ae5-8d3213c86abc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh\" (UID: \"aba8bb4c-a739-480c-9ae5-8d3213c86abc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" Sep 19 10:53:40.215529 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.215500 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9jz4\" (UniqueName: \"kubernetes.io/projected/aba8bb4c-a739-480c-9ae5-8d3213c86abc-kube-api-access-p9jz4\") pod \"payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh\" (UID: \"aba8bb4c-a739-480c-9ae5-8d3213c86abc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" Sep 19 10:53:40.239836 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.238689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" Sep 19 10:53:40.296653 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.296613 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" Sep 19 10:53:40.384809 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.383879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" event={"ID":"c24815e6-1b76-4f80-80bb-f3b5c25c866a","Type":"ContainerStarted","Data":"2bc13e303f0dc54aa8e47b48948f42761fa275482884649f1b791a8b16382419"} Sep 19 10:53:40.384809 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.383942 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" event={"ID":"c24815e6-1b76-4f80-80bb-f3b5c25c866a","Type":"ContainerStarted","Data":"5c27649c6c4fa9210776a0ef97938f1e30dd61ff4833ada1e11a4a2ec8ad5de3"} Sep 19 10:53:40.384809 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.384742 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" Sep 19 10:53:40.536552 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.536480 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" podStartSLOduration=2.536462227 podStartE2EDuration="2.536462227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:53:40.407013387 +0000 UTC m=+1152.319518125" watchObservedRunningTime="2026-09-19 10:53:40.536462227 +0000 UTC m=+1152.448966974" Sep 19 10:53:40.536910 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.536894 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb"] Sep 19 10:53:40.539335 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:53:40.539304 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06314f8b_a1f2_44f9_ab95_52cc7919e6fd.slice/crio-cc682012ab3f8d489a911069eba0fb9c33e8b4352e0d3c6f3da595fb80dfc759 WatchSource:0}: Error finding container cc682012ab3f8d489a911069eba0fb9c33e8b4352e0d3c6f3da595fb80dfc759: Status 404 returned error can't find the container with id cc682012ab3f8d489a911069eba0fb9c33e8b4352e0d3c6f3da595fb80dfc759 Sep 19 10:53:40.628330 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:40.628310 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh"] Sep 19 10:53:40.629312 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:53:40.629291 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba8bb4c_a739_480c_9ae5_8d3213c86abc.slice/crio-e1bf8380f9a64aeebb3fa3f759ef6cf2a2956d324dde47c755b3a59bbcd2fff9 WatchSource:0}: Error finding container e1bf8380f9a64aeebb3fa3f759ef6cf2a2956d324dde47c755b3a59bbcd2fff9: Status 404 returned error can't find the container with id e1bf8380f9a64aeebb3fa3f759ef6cf2a2956d324dde47c755b3a59bbcd2fff9 Sep 19 10:53:41.391854 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:41.391822 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" event={"ID":"aba8bb4c-a739-480c-9ae5-8d3213c86abc","Type":"ContainerStarted","Data":"8796c3b7e0ed715736e1ae3200fbd28d0a54d2d5381ba627e913fe31fe74f68e"} Sep 19 10:53:41.392660 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:41.391893 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" event={"ID":"aba8bb4c-a739-480c-9ae5-8d3213c86abc","Type":"ContainerStarted","Data":"e1bf8380f9a64aeebb3fa3f759ef6cf2a2956d324dde47c755b3a59bbcd2fff9"} Sep 19 10:53:41.392660 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:41.392036 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" Sep 19 10:53:41.393726 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:41.393701 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" event={"ID":"06314f8b-a1f2-44f9-ab95-52cc7919e6fd","Type":"ContainerStarted","Data":"2672ca8a71da71b9d47d2ff7432f9a87d440f15946d9c590ee88d540d464a541"} Sep 19 10:53:41.393876 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:41.393748 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" event={"ID":"06314f8b-a1f2-44f9-ab95-52cc7919e6fd","Type":"ContainerStarted","Data":"cc682012ab3f8d489a911069eba0fb9c33e8b4352e0d3c6f3da595fb80dfc759"} Sep 19 10:53:41.393876 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:41.393856 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" Sep 19 10:53:41.408632 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:41.408597 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" podStartSLOduration=2.408585225 podStartE2EDuration="2.408585225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:53:41.406651531 +0000 UTC m=+1153.319156246" watchObservedRunningTime="2026-09-19 10:53:41.408585225 +0000 UTC m=+1153.321089940" Sep 19 10:53:41.422685 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:41.422636 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" podStartSLOduration=2.422623415 podStartE2EDuration="2.422623415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:53:41.420734713 +0000 UTC m=+1153.333239432" watchObservedRunningTime="2026-09-19 10:53:41.422623415 +0000 UTC m=+1153.335128131" Sep 19 10:53:45.420146 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:45.420117 2579 generic.go:356] "Generic (PLEG): container finished" podID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerID="806fae03de6b58ae179776ea51843ad6b0ab62c54bb3aeb8ee21f52e6b786906" exitCode=0 Sep 19 10:53:45.420533 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:45.420184 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" event={"ID":"2b5096ee-2458-4ac4-9f7c-701756174f18","Type":"ContainerDied","Data":"806fae03de6b58ae179776ea51843ad6b0ab62c54bb3aeb8ee21f52e6b786906"} Sep 19 10:53:45.420610 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:45.420563 2579 scope.go:117] "RemoveContainer" containerID="806fae03de6b58ae179776ea51843ad6b0ab62c54bb3aeb8ee21f52e6b786906" Sep 19 10:53:45.421693 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:45.421662 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:45.421765 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:45.421733 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:45.421802 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:45.421770 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:53:46.428656 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:46.428619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" event={"ID":"2b5096ee-2458-4ac4-9f7c-701756174f18","Type":"ContainerStarted","Data":"43e301298b321935a26450b703c1da20cb6053e6576a68d78c1a71bf9217008a"} Sep 19 10:53:46.582539 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:46.582504 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:46.582776 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:46.582754 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 19 10:53:46.582834 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:46.582800 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 19 10:53:47.583240 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:47.583198 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 19 10:53:47.583633 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:47.583271 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 19 10:53:48.583443 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:48.583405 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 19 10:53:48.583825 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:48.583471 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 19 10:53:49.583352 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:49.583313 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 19 10:53:49.583521 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:49.583365 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 19 10:53:50.583475 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:50.583440 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 19 10:53:50.583647 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:50.583510 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 19 10:53:51.586665 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:51.586637 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:51.587035 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:51.586983 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:51.587642 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:51.587621 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 10:53:52.402141 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:52.402111 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" Sep 19 10:53:52.402318 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:52.402190 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" Sep 19 10:53:52.402373 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:53:52.402321 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" Sep 19 10:54:03.545607 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:03.545571 2579 generic.go:356] "Generic (PLEG): container finished" podID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerID="d87a47f6a811a0dc5f883be378a6c8ecb66974ae8ec23a5a1d4b93260b37309c" exitCode=0 Sep 19 10:54:03.546160 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:03.545648 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" event={"ID":"0c33bf4e-8864-4014-90e5-cb5cf43a07b6","Type":"ContainerDied","Data":"d87a47f6a811a0dc5f883be378a6c8ecb66974ae8ec23a5a1d4b93260b37309c"} Sep 19 10:54:03.546232 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:03.546215 2579 scope.go:117] "RemoveContainer" containerID="d87a47f6a811a0dc5f883be378a6c8ecb66974ae8ec23a5a1d4b93260b37309c" Sep 19 10:54:03.547550 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:03.547521 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:54:03.547625 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:03.547587 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:54:03.547625 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:03.547618 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:54:04.314789 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:04.314755 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:54:04.558073 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:04.558016 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" event={"ID":"0c33bf4e-8864-4014-90e5-cb5cf43a07b6","Type":"ContainerStarted","Data":"4c9790ba95de72d079a9eeb9414dd24b79fb570ea6674d317208976d73a96486"} Sep 19 10:54:04.919408 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:04.919373 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:54:04.920767 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:04.920746 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 19 10:54:04.920826 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:04.920798 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" podUID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 19 10:54:05.920155 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:05.920097 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 19 10:54:05.920530 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:05.920190 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" podUID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 19 10:54:06.920235 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:06.920202 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 19 10:54:06.920611 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:06.920255 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" podUID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 19 10:54:07.919845 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:07.919808 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 19 10:54:07.920021 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:07.919864 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" podUID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 19 10:54:08.919589 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:08.919551 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 19 10:54:08.919981 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:08.919607 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" podUID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 19 10:54:09.924270 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:09.924239 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:54:09.924734 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:09.924653 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:54:09.925266 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:09.925245 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 10:54:14.353485 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.353450 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk"] Sep 19 10:54:14.356818 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.356796 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.359398 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.359380 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e6f44882-openshift-default-dockercfg-fsmgz\"" Sep 19 10:54:14.367382 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.367357 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk"] Sep 19 10:54:14.473979 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.473940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istiod-ca-cert\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.474348 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.474273 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-workload-certs\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.474568 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.474539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwrdr\" (UniqueName: \"kubernetes.io/projected/2b3a8639-373e-4d27-83c2-02eefc29d0cd-kube-api-access-mwrdr\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.474709 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.474625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-podinfo\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.474709 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.474670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-data\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.474709 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.474698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-envoy\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.474896 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.474748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-workload-socket\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.474896 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.474779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-token\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.474896 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.474804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-credential-socket\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.576115 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.576079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-podinfo\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.576299 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.576133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-data\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.576299 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.576161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-envoy\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.576299 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.576203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-workload-socket\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.576299 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.576232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-token\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.576299 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.576256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-credential-socket\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.576585 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.576327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istiod-ca-cert\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.576585 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.576460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-workload-certs\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.576585 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.576526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrdr\" (UniqueName: \"kubernetes.io/projected/2b3a8639-373e-4d27-83c2-02eefc29d0cd-kube-api-access-mwrdr\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.576585 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.576578 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-data\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.576986 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.576881 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-credential-socket\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.576986 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.576929 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-workload-socket\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.577185 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.577009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-workload-certs\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.577185 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.577105 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istiod-ca-cert\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.578640 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.578618 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-envoy\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.578748 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.578732 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-podinfo\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.583586 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.583565 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-token\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.583842 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.583823 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrdr\" (UniqueName: \"kubernetes.io/projected/2b3a8639-373e-4d27-83c2-02eefc29d0cd-kube-api-access-mwrdr\") pod \"e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.669106 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.669071 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:14.945391 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.945366 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk"] Sep 19 10:54:14.947754 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:54:14.947721 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3a8639_373e_4d27_83c2_02eefc29d0cd.slice/crio-9509e705aa87beca1155e78d8df7f8bb24847e0127eb64cab61f97b3c1d3bcb7 WatchSource:0}: Error finding container 9509e705aa87beca1155e78d8df7f8bb24847e0127eb64cab61f97b3c1d3bcb7: Status 404 returned error can't find the container with id 9509e705aa87beca1155e78d8df7f8bb24847e0127eb64cab61f97b3c1d3bcb7 Sep 19 10:54:14.950216 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.950186 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:54:14.950290 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.950251 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:54:14.950290 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:14.950278 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:54:15.629978 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:15.629029 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" event={"ID":"2b3a8639-373e-4d27-83c2-02eefc29d0cd","Type":"ContainerStarted","Data":"c4457ce61563ef951b3ecb847fcf059a7a1640b4198b6347739eac66da43045e"} Sep 19 10:54:15.629978 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:15.629118 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" event={"ID":"2b3a8639-373e-4d27-83c2-02eefc29d0cd","Type":"ContainerStarted","Data":"9509e705aa87beca1155e78d8df7f8bb24847e0127eb64cab61f97b3c1d3bcb7"} Sep 19 10:54:15.649905 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:15.649841 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" podStartSLOduration=1.649817622 podStartE2EDuration="1.649817622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:54:15.647721968 +0000 UTC m=+1187.560226686" watchObservedRunningTime="2026-09-19 10:54:15.649817622 +0000 UTC m=+1187.562322342" Sep 19 10:54:15.669349 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:15.669315 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:16.674776 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:16.674746 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:17.641404 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:17.641369 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:17.642461 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:17.642435 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:30.253642 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:30.253610 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 10:54:30.254186 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:30.254169 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 10:54:30.336858 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:30.330922 2579 scope.go:117] "RemoveContainer" containerID="64b6995452fbeb9407b55a60bb45e30aca7c4d42bf2366f9d0062b6da004dfd5" Sep 19 10:54:30.345800 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:30.345777 2579 scope.go:117] "RemoveContainer" containerID="575e52a79f1f70337b626cf28f6869f08a8bc46d6d5b8e4b849822cc274e5ae2" Sep 19 10:54:48.683226 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:48.683184 2579 patch_prober.go:28] interesting pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 19 10:54:48.683660 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:48.683237 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 19 10:54:49.845762 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:49.845673 2579 generic.go:356] "Generic (PLEG): container finished" podID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerID="c4457ce61563ef951b3ecb847fcf059a7a1640b4198b6347739eac66da43045e" exitCode=0 Sep 19 10:54:49.846184 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:49.845755 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" event={"ID":"2b3a8639-373e-4d27-83c2-02eefc29d0cd","Type":"ContainerDied","Data":"c4457ce61563ef951b3ecb847fcf059a7a1640b4198b6347739eac66da43045e"} Sep 19 10:54:49.846236 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:49.846200 2579 scope.go:117] "RemoveContainer" containerID="c4457ce61563ef951b3ecb847fcf059a7a1640b4198b6347739eac66da43045e" Sep 19 10:54:49.847409 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:49.847384 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:54:49.847494 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:49.847459 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:54:49.847494 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:49.847488 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:54:50.853946 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:50.853908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" event={"ID":"2b3a8639-373e-4d27-83c2-02eefc29d0cd","Type":"ContainerStarted","Data":"f04bdaf803bdf2487febec117c2e3ade0ad9dc8a45a9a4a59c29a72273e1aacf"} Sep 19 10:54:51.669905 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:51.669866 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:51.670291 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:51.670265 2579 patch_prober.go:28] interesting pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 19 10:54:51.670354 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:51.670317 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 19 10:54:52.669986 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:52.669952 2579 patch_prober.go:28] interesting pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 19 10:54:52.670382 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:52.670004 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 19 10:54:53.669773 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:53.669737 2579 patch_prober.go:28] interesting pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 19 10:54:53.669975 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:53.669794 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 19 10:54:54.670132 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:54.670092 2579 patch_prober.go:28] interesting pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 19 10:54:54.670509 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:54.670156 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 19 10:54:55.670305 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:55.670271 2579 patch_prober.go:28] interesting pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 19 10:54:55.670696 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:55.670338 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 19 10:54:56.670124 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:56.670086 2579 patch_prober.go:28] interesting pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 19 10:54:56.670304 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:56.670153 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 19 10:54:57.674356 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:57.674324 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:57.674841 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:57.674655 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:54:57.675257 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:54:57.675235 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:55:17.849171 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:17.849140 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk"] Sep 19 10:55:17.856688 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:17.854901 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk" Sep 19 10:55:17.857962 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:17.857911 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 10:55:17.858386 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:17.858338 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 10:55:17.861814 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:17.861794 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk"] Sep 19 10:55:17.919912 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:17.919879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819f9d58-61b9-46fa-b3b3-1aa212be7a01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk\" (UID: \"819f9d58-61b9-46fa-b3b3-1aa212be7a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk" Sep 19 10:55:18.020424 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:18.020392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819f9d58-61b9-46fa-b3b3-1aa212be7a01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk\" (UID: \"819f9d58-61b9-46fa-b3b3-1aa212be7a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk" Sep 19 10:55:18.020993 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:18.020977 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819f9d58-61b9-46fa-b3b3-1aa212be7a01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk\" (UID: \"819f9d58-61b9-46fa-b3b3-1aa212be7a01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk" Sep 19 10:55:18.169918 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:18.169880 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk" Sep 19 10:55:18.445138 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:18.445109 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk"] Sep 19 10:55:18.445683 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:55:18.445657 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819f9d58_61b9_46fa_b3b3_1aa212be7a01.slice/crio-6a42dbcca503bc12c70ef7b6ec8f5711530943c871d9ea96625494e1aa0700c6 WatchSource:0}: Error finding container 6a42dbcca503bc12c70ef7b6ec8f5711530943c871d9ea96625494e1aa0700c6: Status 404 returned error can't find the container with id 6a42dbcca503bc12c70ef7b6ec8f5711530943c871d9ea96625494e1aa0700c6 Sep 19 10:55:19.020743 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:19.020665 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk" event={"ID":"819f9d58-61b9-46fa-b3b3-1aa212be7a01","Type":"ContainerStarted","Data":"b2dbe0f8039fd3bf09e0474934a6d0048bde26b89cdfa37317f692480b09cfec"} Sep 19 10:55:19.020743 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:19.020716 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk" event={"ID":"819f9d58-61b9-46fa-b3b3-1aa212be7a01","Type":"ContainerStarted","Data":"6a42dbcca503bc12c70ef7b6ec8f5711530943c871d9ea96625494e1aa0700c6"} Sep 19 10:55:19.035777 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:19.035734 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk" podStartSLOduration=2.035700627 podStartE2EDuration="2.035700627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:55:19.035229844 +0000 UTC m=+1250.947734561" watchObservedRunningTime="2026-09-19 10:55:19.035700627 +0000 UTC m=+1250.948205361" Sep 19 10:55:20.028050 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:20.028009 2579 generic.go:356] "Generic (PLEG): container finished" podID="819f9d58-61b9-46fa-b3b3-1aa212be7a01" containerID="b2dbe0f8039fd3bf09e0474934a6d0048bde26b89cdfa37317f692480b09cfec" exitCode=0 Sep 19 10:55:20.028480 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:20.028109 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk" event={"ID":"819f9d58-61b9-46fa-b3b3-1aa212be7a01","Type":"ContainerDied","Data":"b2dbe0f8039fd3bf09e0474934a6d0048bde26b89cdfa37317f692480b09cfec"} Sep 19 10:55:21.179098 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:21.179054 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk" Sep 19 10:55:21.250202 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:21.250172 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819f9d58-61b9-46fa-b3b3-1aa212be7a01-maas-api-service-ca\") pod \"819f9d58-61b9-46fa-b3b3-1aa212be7a01\" (UID: \"819f9d58-61b9-46fa-b3b3-1aa212be7a01\") " Sep 19 10:55:21.250534 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:21.250512 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819f9d58-61b9-46fa-b3b3-1aa212be7a01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "819f9d58-61b9-46fa-b3b3-1aa212be7a01" (UID: "819f9d58-61b9-46fa-b3b3-1aa212be7a01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:55:21.351159 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:21.351092 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819f9d58-61b9-46fa-b3b3-1aa212be7a01-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:22.042000 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:22.041966 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk" Sep 19 10:55:22.042000 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:22.041990 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk" event={"ID":"819f9d58-61b9-46fa-b3b3-1aa212be7a01","Type":"ContainerDied","Data":"6a42dbcca503bc12c70ef7b6ec8f5711530943c871d9ea96625494e1aa0700c6"} Sep 19 10:55:22.042224 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:22.042034 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a42dbcca503bc12c70ef7b6ec8f5711530943c871d9ea96625494e1aa0700c6" Sep 19 10:55:22.728153 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:22.728118 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr"] Sep 19 10:55:30.371249 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:30.371206 2579 scope.go:117] "RemoveContainer" containerID="a4d7bc7b772a8008a7262ea34afadc7c8963d16156b5dc3a33107a6603b721a2" Sep 19 10:55:30.379916 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:30.379896 2579 scope.go:117] "RemoveContainer" containerID="285f0deda549e6e0f41c584f354048acd89db896a43f990b0feb42fff9e940d7" Sep 19 10:55:33.468456 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.468421 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv"] Sep 19 10:55:33.470092 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.470027 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" podUID="8bcec4da-5b43-48e3-bfed-33511225d966" containerName="payload-processing" containerID="cri-o://99f975e75903d302bbd6b905a138d0cf52c0050b014a6c1fec1c6213c49bd9d5" gracePeriod=30 Sep 19 10:55:33.472577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.472544 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk"] Sep 19 10:55:33.472767 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.472731 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" podUID="0d62ccc5-db7b-4638-bbf1-4e7297c0f68a" containerName="payload-pre-processing" containerID="cri-o://75426e78207cfbc848158df6d0d695928cab3522aa326e2f752712e97db4e75f" gracePeriod=30 Sep 19 10:55:33.620337 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.620299 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk"] Sep 19 10:55:33.624691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.624187 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-580e01-vxxxk"] Sep 19 10:55:33.792978 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.792954 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" Sep 19 10:55:33.796602 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.796575 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" Sep 19 10:55:33.870977 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.870944 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbl4f\" (UniqueName: \"kubernetes.io/projected/8bcec4da-5b43-48e3-bfed-33511225d966-kube-api-access-vbl4f\") pod \"8bcec4da-5b43-48e3-bfed-33511225d966\" (UID: \"8bcec4da-5b43-48e3-bfed-33511225d966\") " Sep 19 10:55:33.871160 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.871019 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bcec4da-5b43-48e3-bfed-33511225d966-plugins-config-volume\") pod \"8bcec4da-5b43-48e3-bfed-33511225d966\" (UID: \"8bcec4da-5b43-48e3-bfed-33511225d966\") " Sep 19 10:55:33.871160 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.871048 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz75k\" (UniqueName: \"kubernetes.io/projected/0d62ccc5-db7b-4638-bbf1-4e7297c0f68a-kube-api-access-nz75k\") pod \"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a\" (UID: \"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a\") " Sep 19 10:55:33.871160 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.871104 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d62ccc5-db7b-4638-bbf1-4e7297c0f68a-plugins-config-volume\") pod \"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a\" (UID: \"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a\") " Sep 19 10:55:33.871477 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.871452 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcec4da-5b43-48e3-bfed-33511225d966-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8bcec4da-5b43-48e3-bfed-33511225d966" (UID: "8bcec4da-5b43-48e3-bfed-33511225d966"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:55:33.871552 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.871482 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d62ccc5-db7b-4638-bbf1-4e7297c0f68a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d62ccc5-db7b-4638-bbf1-4e7297c0f68a" (UID: "0d62ccc5-db7b-4638-bbf1-4e7297c0f68a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:55:33.871595 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.871576 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bcec4da-5b43-48e3-bfed-33511225d966-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:33.871595 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.871589 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d62ccc5-db7b-4638-bbf1-4e7297c0f68a-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:33.873156 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.873134 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcec4da-5b43-48e3-bfed-33511225d966-kube-api-access-vbl4f" (OuterVolumeSpecName: "kube-api-access-vbl4f") pod "8bcec4da-5b43-48e3-bfed-33511225d966" (UID: "8bcec4da-5b43-48e3-bfed-33511225d966"). InnerVolumeSpecName "kube-api-access-vbl4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:55:33.873213 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.873159 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d62ccc5-db7b-4638-bbf1-4e7297c0f68a-kube-api-access-nz75k" (OuterVolumeSpecName: "kube-api-access-nz75k") pod "0d62ccc5-db7b-4638-bbf1-4e7297c0f68a" (UID: "0d62ccc5-db7b-4638-bbf1-4e7297c0f68a"). InnerVolumeSpecName "kube-api-access-nz75k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:55:33.972888 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.972846 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz75k\" (UniqueName: \"kubernetes.io/projected/0d62ccc5-db7b-4638-bbf1-4e7297c0f68a-kube-api-access-nz75k\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:33.972888 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:33.972880 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbl4f\" (UniqueName: \"kubernetes.io/projected/8bcec4da-5b43-48e3-bfed-33511225d966-kube-api-access-vbl4f\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:34.116957 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.116862 2579 generic.go:356] "Generic (PLEG): container finished" podID="8bcec4da-5b43-48e3-bfed-33511225d966" containerID="99f975e75903d302bbd6b905a138d0cf52c0050b014a6c1fec1c6213c49bd9d5" exitCode=0 Sep 19 10:55:34.116957 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.116913 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" Sep 19 10:55:34.117213 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.116953 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" event={"ID":"8bcec4da-5b43-48e3-bfed-33511225d966","Type":"ContainerDied","Data":"99f975e75903d302bbd6b905a138d0cf52c0050b014a6c1fec1c6213c49bd9d5"} Sep 19 10:55:34.117213 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.117015 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv" event={"ID":"8bcec4da-5b43-48e3-bfed-33511225d966","Type":"ContainerDied","Data":"4cf2f3d47e1b0234d2033a21d477a85d819b885f6b66f00d92c5863a8989b079"} Sep 19 10:55:34.117213 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.117044 2579 scope.go:117] "RemoveContainer" containerID="99f975e75903d302bbd6b905a138d0cf52c0050b014a6c1fec1c6213c49bd9d5" Sep 19 10:55:34.118356 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.118332 2579 generic.go:356] "Generic (PLEG): container finished" podID="0d62ccc5-db7b-4638-bbf1-4e7297c0f68a" containerID="75426e78207cfbc848158df6d0d695928cab3522aa326e2f752712e97db4e75f" exitCode=0 Sep 19 10:55:34.118481 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.118366 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" Sep 19 10:55:34.118481 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.118409 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" event={"ID":"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a","Type":"ContainerDied","Data":"75426e78207cfbc848158df6d0d695928cab3522aa326e2f752712e97db4e75f"} Sep 19 10:55:34.118481 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.118457 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk" event={"ID":"0d62ccc5-db7b-4638-bbf1-4e7297c0f68a","Type":"ContainerDied","Data":"2aeb48f56bc5b0f931698352f4312e12b7b7a77a42a0635f0701c4c57cc12762"} Sep 19 10:55:34.127153 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.127135 2579 scope.go:117] "RemoveContainer" containerID="99f975e75903d302bbd6b905a138d0cf52c0050b014a6c1fec1c6213c49bd9d5" Sep 19 10:55:34.127425 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:55:34.127407 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f975e75903d302bbd6b905a138d0cf52c0050b014a6c1fec1c6213c49bd9d5\": container with ID starting with 99f975e75903d302bbd6b905a138d0cf52c0050b014a6c1fec1c6213c49bd9d5 not found: ID does not exist" containerID="99f975e75903d302bbd6b905a138d0cf52c0050b014a6c1fec1c6213c49bd9d5" Sep 19 10:55:34.127508 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.127437 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f975e75903d302bbd6b905a138d0cf52c0050b014a6c1fec1c6213c49bd9d5"} err="failed to get container status \"99f975e75903d302bbd6b905a138d0cf52c0050b014a6c1fec1c6213c49bd9d5\": rpc error: code = NotFound desc = could not find container \"99f975e75903d302bbd6b905a138d0cf52c0050b014a6c1fec1c6213c49bd9d5\": container with ID starting with 99f975e75903d302bbd6b905a138d0cf52c0050b014a6c1fec1c6213c49bd9d5 not found: ID does not exist" Sep 19 10:55:34.127508 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.127461 2579 scope.go:117] "RemoveContainer" containerID="75426e78207cfbc848158df6d0d695928cab3522aa326e2f752712e97db4e75f" Sep 19 10:55:34.144257 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.139576 2579 scope.go:117] "RemoveContainer" containerID="75426e78207cfbc848158df6d0d695928cab3522aa326e2f752712e97db4e75f" Sep 19 10:55:34.144257 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:55:34.140246 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75426e78207cfbc848158df6d0d695928cab3522aa326e2f752712e97db4e75f\": container with ID starting with 75426e78207cfbc848158df6d0d695928cab3522aa326e2f752712e97db4e75f not found: ID does not exist" containerID="75426e78207cfbc848158df6d0d695928cab3522aa326e2f752712e97db4e75f" Sep 19 10:55:34.144257 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.140277 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75426e78207cfbc848158df6d0d695928cab3522aa326e2f752712e97db4e75f"} err="failed to get container status \"75426e78207cfbc848158df6d0d695928cab3522aa326e2f752712e97db4e75f\": rpc error: code = NotFound desc = could not find container \"75426e78207cfbc848158df6d0d695928cab3522aa326e2f752712e97db4e75f\": container with ID starting with 75426e78207cfbc848158df6d0d695928cab3522aa326e2f752712e97db4e75f not found: ID does not exist" Sep 19 10:55:34.148792 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.148767 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv"] Sep 19 10:55:34.152878 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.152851 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv"] Sep 19 10:55:34.156942 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.156921 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk"] Sep 19 10:55:34.161868 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.161847 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk"] Sep 19 10:55:34.196169 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.196147 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d62ccc5-db7b-4638-bbf1-4e7297c0f68a" path="/var/lib/kubelet/pods/0d62ccc5-db7b-4638-bbf1-4e7297c0f68a/volumes" Sep 19 10:55:34.196508 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.196495 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="819f9d58-61b9-46fa-b3b3-1aa212be7a01" path="/var/lib/kubelet/pods/819f9d58-61b9-46fa-b3b3-1aa212be7a01/volumes" Sep 19 10:55:34.196840 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:34.196828 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bcec4da-5b43-48e3-bfed-33511225d966" path="/var/lib/kubelet/pods/8bcec4da-5b43-48e3-bfed-33511225d966/volumes" Sep 19 10:55:37.749080 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:37.749016 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" podUID="216cf07a-3add-4258-820b-1be053bd9a0f" containerName="main" containerID="cri-o://80392e04c8a94d8af21c82e356af7341db5c3d9fe78f1092ce69d6b7060f36e4" gracePeriod=45 Sep 19 10:55:38.006791 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.006735 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:55:38.116444 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.116414 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-home\") pod \"216cf07a-3add-4258-820b-1be053bd9a0f\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " Sep 19 10:55:38.116958 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.116478 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cpxr\" (UniqueName: \"kubernetes.io/projected/216cf07a-3add-4258-820b-1be053bd9a0f-kube-api-access-8cpxr\") pod \"216cf07a-3add-4258-820b-1be053bd9a0f\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " Sep 19 10:55:38.116958 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.116535 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-model-cache\") pod \"216cf07a-3add-4258-820b-1be053bd9a0f\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " Sep 19 10:55:38.116958 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.116655 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-dshm\") pod \"216cf07a-3add-4258-820b-1be053bd9a0f\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " Sep 19 10:55:38.116958 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.116678 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-tmp-dir\") pod \"216cf07a-3add-4258-820b-1be053bd9a0f\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " Sep 19 10:55:38.116958 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.116710 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216cf07a-3add-4258-820b-1be053bd9a0f-tls-certs\") pod \"216cf07a-3add-4258-820b-1be053bd9a0f\" (UID: \"216cf07a-3add-4258-820b-1be053bd9a0f\") " Sep 19 10:55:38.116958 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.116713 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-home" (OuterVolumeSpecName: "home") pod "216cf07a-3add-4258-820b-1be053bd9a0f" (UID: "216cf07a-3add-4258-820b-1be053bd9a0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:55:38.116958 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.116861 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "216cf07a-3add-4258-820b-1be053bd9a0f" (UID: "216cf07a-3add-4258-820b-1be053bd9a0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:55:38.117417 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.117177 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "216cf07a-3add-4258-820b-1be053bd9a0f" (UID: "216cf07a-3add-4258-820b-1be053bd9a0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:55:38.117768 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.117741 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:38.117855 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.117772 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:38.117855 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.117787 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:38.119199 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.119174 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216cf07a-3add-4258-820b-1be053bd9a0f-kube-api-access-8cpxr" (OuterVolumeSpecName: "kube-api-access-8cpxr") pod "216cf07a-3add-4258-820b-1be053bd9a0f" (UID: "216cf07a-3add-4258-820b-1be053bd9a0f"). InnerVolumeSpecName "kube-api-access-8cpxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:55:38.119322 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.119304 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216cf07a-3add-4258-820b-1be053bd9a0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "216cf07a-3add-4258-820b-1be053bd9a0f" (UID: "216cf07a-3add-4258-820b-1be053bd9a0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 10:55:38.119372 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.119306 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-dshm" (OuterVolumeSpecName: "dshm") pod "216cf07a-3add-4258-820b-1be053bd9a0f" (UID: "216cf07a-3add-4258-820b-1be053bd9a0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:55:38.149936 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.149903 2579 generic.go:356] "Generic (PLEG): container finished" podID="216cf07a-3add-4258-820b-1be053bd9a0f" containerID="80392e04c8a94d8af21c82e356af7341db5c3d9fe78f1092ce69d6b7060f36e4" exitCode=0 Sep 19 10:55:38.150054 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.149947 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" Sep 19 10:55:38.150054 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.149985 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" event={"ID":"216cf07a-3add-4258-820b-1be053bd9a0f","Type":"ContainerDied","Data":"80392e04c8a94d8af21c82e356af7341db5c3d9fe78f1092ce69d6b7060f36e4"} Sep 19 10:55:38.150054 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.150033 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr" event={"ID":"216cf07a-3add-4258-820b-1be053bd9a0f","Type":"ContainerDied","Data":"c9a20348e4b81ff3fde0fad2754cd66f3370a74fe336563955c6afc80d2e33e1"} Sep 19 10:55:38.150054 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.150052 2579 scope.go:117] "RemoveContainer" containerID="80392e04c8a94d8af21c82e356af7341db5c3d9fe78f1092ce69d6b7060f36e4" Sep 19 10:55:38.159392 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.159376 2579 scope.go:117] "RemoveContainer" containerID="80392e04c8a94d8af21c82e356af7341db5c3d9fe78f1092ce69d6b7060f36e4" Sep 19 10:55:38.159702 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:55:38.159681 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80392e04c8a94d8af21c82e356af7341db5c3d9fe78f1092ce69d6b7060f36e4\": container with ID starting with 80392e04c8a94d8af21c82e356af7341db5c3d9fe78f1092ce69d6b7060f36e4 not found: ID does not exist" containerID="80392e04c8a94d8af21c82e356af7341db5c3d9fe78f1092ce69d6b7060f36e4" Sep 19 10:55:38.159769 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.159709 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80392e04c8a94d8af21c82e356af7341db5c3d9fe78f1092ce69d6b7060f36e4"} err="failed to get container status \"80392e04c8a94d8af21c82e356af7341db5c3d9fe78f1092ce69d6b7060f36e4\": rpc error: code = NotFound desc = could not find container \"80392e04c8a94d8af21c82e356af7341db5c3d9fe78f1092ce69d6b7060f36e4\": container with ID starting with 80392e04c8a94d8af21c82e356af7341db5c3d9fe78f1092ce69d6b7060f36e4 not found: ID does not exist" Sep 19 10:55:38.170161 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.170141 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr"] Sep 19 10:55:38.176404 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.176383 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-580e01/distinct-w3-580e01-kserve-7c5d94d8f-kw5zr"] Sep 19 10:55:38.196290 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.196265 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="216cf07a-3add-4258-820b-1be053bd9a0f" path="/var/lib/kubelet/pods/216cf07a-3add-4258-820b-1be053bd9a0f/volumes" Sep 19 10:55:38.219497 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.218908 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216cf07a-3add-4258-820b-1be053bd9a0f-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:38.219497 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.219378 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216cf07a-3add-4258-820b-1be053bd9a0f-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:38.219497 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:38.219468 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cpxr\" (UniqueName: \"kubernetes.io/projected/216cf07a-3add-4258-820b-1be053bd9a0f-kube-api-access-8cpxr\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:40.130288 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:40.130252 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt"] Sep 19 10:55:40.130760 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:40.130507 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" containerID="cri-o://a8e85408b73a35af91294e8b838e5af718b9e177dabd6d5e9304d867acd39cc4" gracePeriod=30 Sep 19 10:55:43.446102 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:43.446034 2579 patch_prober.go:28] interesting pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 19 10:55:43.446464 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:43.446135 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 10:55:45.696781 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.696754 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:55:45.805789 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.805713 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6939ddf4-8535-4764-ac63-12646e8a14bb-istiod-ca-cert\") pod \"6939ddf4-8535-4764-ac63-12646e8a14bb\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " Sep 19 10:55:45.805789 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.805775 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-workload-certs\") pod \"6939ddf4-8535-4764-ac63-12646e8a14bb\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " Sep 19 10:55:45.806019 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.805794 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-credential-socket\") pod \"6939ddf4-8535-4764-ac63-12646e8a14bb\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " Sep 19 10:55:45.806019 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.805813 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdp8d\" (UniqueName: \"kubernetes.io/projected/6939ddf4-8535-4764-ac63-12646e8a14bb-kube-api-access-mdp8d\") pod \"6939ddf4-8535-4764-ac63-12646e8a14bb\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " Sep 19 10:55:45.806019 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.805853 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-token\") pod \"6939ddf4-8535-4764-ac63-12646e8a14bb\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " Sep 19 10:55:45.806019 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.805867 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-envoy\") pod \"6939ddf4-8535-4764-ac63-12646e8a14bb\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " Sep 19 10:55:45.806019 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.805908 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-workload-socket\") pod \"6939ddf4-8535-4764-ac63-12646e8a14bb\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " Sep 19 10:55:45.806019 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.805950 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-podinfo\") pod \"6939ddf4-8535-4764-ac63-12646e8a14bb\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " Sep 19 10:55:45.806019 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.806019 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-data\") pod \"6939ddf4-8535-4764-ac63-12646e8a14bb\" (UID: \"6939ddf4-8535-4764-ac63-12646e8a14bb\") " Sep 19 10:55:45.806391 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.806044 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6939ddf4-8535-4764-ac63-12646e8a14bb" (UID: "6939ddf4-8535-4764-ac63-12646e8a14bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:55:45.806391 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.806099 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6939ddf4-8535-4764-ac63-12646e8a14bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6939ddf4-8535-4764-ac63-12646e8a14bb" (UID: "6939ddf4-8535-4764-ac63-12646e8a14bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:55:45.806391 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.806194 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6939ddf4-8535-4764-ac63-12646e8a14bb" (UID: "6939ddf4-8535-4764-ac63-12646e8a14bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:55:45.806577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.806414 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6939ddf4-8535-4764-ac63-12646e8a14bb" (UID: "6939ddf4-8535-4764-ac63-12646e8a14bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:55:45.806577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.806429 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6939ddf4-8535-4764-ac63-12646e8a14bb-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:45.806577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.806438 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "6939ddf4-8535-4764-ac63-12646e8a14bb" (UID: "6939ddf4-8535-4764-ac63-12646e8a14bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:55:45.806577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.806451 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:45.806577 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.806464 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:45.808288 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.808258 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6939ddf4-8535-4764-ac63-12646e8a14bb" (UID: "6939ddf4-8535-4764-ac63-12646e8a14bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 10:55:45.808391 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.808347 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6939ddf4-8535-4764-ac63-12646e8a14bb" (UID: "6939ddf4-8535-4764-ac63-12646e8a14bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:55:45.808391 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.808357 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "6939ddf4-8535-4764-ac63-12646e8a14bb" (UID: "6939ddf4-8535-4764-ac63-12646e8a14bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:55:45.808475 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.808444 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6939ddf4-8535-4764-ac63-12646e8a14bb-kube-api-access-mdp8d" (OuterVolumeSpecName: "kube-api-access-mdp8d") pod "6939ddf4-8535-4764-ac63-12646e8a14bb" (UID: "6939ddf4-8535-4764-ac63-12646e8a14bb"). InnerVolumeSpecName "kube-api-access-mdp8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:55:45.907674 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.907639 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:45.907674 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.907668 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdp8d\" (UniqueName: \"kubernetes.io/projected/6939ddf4-8535-4764-ac63-12646e8a14bb-kube-api-access-mdp8d\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:45.907674 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.907679 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:45.907913 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.907688 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:45.907913 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.907695 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6939ddf4-8535-4764-ac63-12646e8a14bb-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:45.907913 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:45.907703 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6939ddf4-8535-4764-ac63-12646e8a14bb-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:55:46.198158 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:46.198124 2579 generic.go:356] "Generic (PLEG): container finished" podID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerID="a8e85408b73a35af91294e8b838e5af718b9e177dabd6d5e9304d867acd39cc4" exitCode=0 Sep 19 10:55:46.198158 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:46.198157 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" Sep 19 10:55:46.198376 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:46.198202 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" event={"ID":"6939ddf4-8535-4764-ac63-12646e8a14bb","Type":"ContainerDied","Data":"a8e85408b73a35af91294e8b838e5af718b9e177dabd6d5e9304d867acd39cc4"} Sep 19 10:55:46.198376 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:46.198244 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt" event={"ID":"6939ddf4-8535-4764-ac63-12646e8a14bb","Type":"ContainerDied","Data":"04ed59e988abd97c79412e1550cd2994c31b809bde4564bee3fe62990a0f7fc1"} Sep 19 10:55:46.198376 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:46.198271 2579 scope.go:117] "RemoveContainer" containerID="a8e85408b73a35af91294e8b838e5af718b9e177dabd6d5e9304d867acd39cc4" Sep 19 10:55:46.208518 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:46.208491 2579 scope.go:117] "RemoveContainer" containerID="e2ed3960a1cd134414eae7dc1de2e3e911481d79bff8f085da28ff3680839b0e" Sep 19 10:55:46.216674 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:46.216656 2579 scope.go:117] "RemoveContainer" containerID="a8e85408b73a35af91294e8b838e5af718b9e177dabd6d5e9304d867acd39cc4" Sep 19 10:55:46.216916 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:55:46.216896 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e85408b73a35af91294e8b838e5af718b9e177dabd6d5e9304d867acd39cc4\": container with ID starting with a8e85408b73a35af91294e8b838e5af718b9e177dabd6d5e9304d867acd39cc4 not found: ID does not exist" containerID="a8e85408b73a35af91294e8b838e5af718b9e177dabd6d5e9304d867acd39cc4" Sep 19 10:55:46.216982 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:46.216923 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e85408b73a35af91294e8b838e5af718b9e177dabd6d5e9304d867acd39cc4"} err="failed to get container status \"a8e85408b73a35af91294e8b838e5af718b9e177dabd6d5e9304d867acd39cc4\": rpc error: code = NotFound desc = could not find container \"a8e85408b73a35af91294e8b838e5af718b9e177dabd6d5e9304d867acd39cc4\": container with ID starting with a8e85408b73a35af91294e8b838e5af718b9e177dabd6d5e9304d867acd39cc4 not found: ID does not exist" Sep 19 10:55:46.216982 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:46.216940 2579 scope.go:117] "RemoveContainer" containerID="e2ed3960a1cd134414eae7dc1de2e3e911481d79bff8f085da28ff3680839b0e" Sep 19 10:55:46.217208 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:55:46.217192 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ed3960a1cd134414eae7dc1de2e3e911481d79bff8f085da28ff3680839b0e\": container with ID starting with e2ed3960a1cd134414eae7dc1de2e3e911481d79bff8f085da28ff3680839b0e not found: ID does not exist" containerID="e2ed3960a1cd134414eae7dc1de2e3e911481d79bff8f085da28ff3680839b0e" Sep 19 10:55:46.217255 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:46.217215 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ed3960a1cd134414eae7dc1de2e3e911481d79bff8f085da28ff3680839b0e"} err="failed to get container status \"e2ed3960a1cd134414eae7dc1de2e3e911481d79bff8f085da28ff3680839b0e\": rpc error: code = NotFound desc = could not find container \"e2ed3960a1cd134414eae7dc1de2e3e911481d79bff8f085da28ff3680839b0e\": container with ID starting with e2ed3960a1cd134414eae7dc1de2e3e911481d79bff8f085da28ff3680839b0e not found: ID does not exist" Sep 19 10:55:46.221210 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:46.221190 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt"] Sep 19 10:55:46.232796 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:46.230285 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt"] Sep 19 10:55:48.196972 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:55:48.196930 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" path="/var/lib/kubelet/pods/6939ddf4-8535-4764-ac63-12646e8a14bb/volumes" Sep 19 10:56:13.614813 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.614775 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs"] Sep 19 10:56:13.615284 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615253 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="216cf07a-3add-4258-820b-1be053bd9a0f" containerName="main" Sep 19 10:56:13.615284 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615265 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="216cf07a-3add-4258-820b-1be053bd9a0f" containerName="main" Sep 19 10:56:13.615284 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615280 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" Sep 19 10:56:13.615284 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615285 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" Sep 19 10:56:13.615421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615293 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bcec4da-5b43-48e3-bfed-33511225d966" containerName="payload-processing" Sep 19 10:56:13.615421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615299 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bcec4da-5b43-48e3-bfed-33511225d966" containerName="payload-processing" Sep 19 10:56:13.615421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615309 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" Sep 19 10:56:13.615421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615314 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" Sep 19 10:56:13.615421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615329 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d62ccc5-db7b-4638-bbf1-4e7297c0f68a" containerName="payload-pre-processing" Sep 19 10:56:13.615421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615335 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d62ccc5-db7b-4638-bbf1-4e7297c0f68a" containerName="payload-pre-processing" Sep 19 10:56:13.615421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615347 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="819f9d58-61b9-46fa-b3b3-1aa212be7a01" containerName="revoke-keys" Sep 19 10:56:13.615421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615352 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="819f9d58-61b9-46fa-b3b3-1aa212be7a01" containerName="revoke-keys" Sep 19 10:56:13.615421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615405 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" Sep 19 10:56:13.615421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615413 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="819f9d58-61b9-46fa-b3b3-1aa212be7a01" containerName="revoke-keys" Sep 19 10:56:13.615421 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615420 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="216cf07a-3add-4258-820b-1be053bd9a0f" containerName="main" Sep 19 10:56:13.615749 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615427 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d62ccc5-db7b-4638-bbf1-4e7297c0f68a" containerName="payload-pre-processing" Sep 19 10:56:13.615749 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.615433 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bcec4da-5b43-48e3-bfed-33511225d966" containerName="payload-processing" Sep 19 10:56:13.618618 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.618600 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs" Sep 19 10:56:13.620868 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.620839 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 10:56:13.620989 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.620909 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 10:56:13.627790 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.627765 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs"] Sep 19 10:56:13.664510 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.664480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82ed89d-c83d-4146-91c7-30dbd3cd1d66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs\" (UID: \"a82ed89d-c83d-4146-91c7-30dbd3cd1d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs" Sep 19 10:56:13.765842 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.765815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82ed89d-c83d-4146-91c7-30dbd3cd1d66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs\" (UID: \"a82ed89d-c83d-4146-91c7-30dbd3cd1d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs" Sep 19 10:56:13.766401 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.766383 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82ed89d-c83d-4146-91c7-30dbd3cd1d66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs\" (UID: \"a82ed89d-c83d-4146-91c7-30dbd3cd1d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs" Sep 19 10:56:13.930054 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:13.930027 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs" Sep 19 10:56:14.200911 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:14.200880 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs"] Sep 19 10:56:14.203312 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:56:14.203287 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82ed89d_c83d_4146_91c7_30dbd3cd1d66.slice/crio-cd7ca7be17956d263abde1889e36d0b6348d127cfc32e8c6b3faca78b9c85010 WatchSource:0}: Error finding container cd7ca7be17956d263abde1889e36d0b6348d127cfc32e8c6b3faca78b9c85010: Status 404 returned error can't find the container with id cd7ca7be17956d263abde1889e36d0b6348d127cfc32e8c6b3faca78b9c85010 Sep 19 10:56:14.366408 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:14.366382 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs" event={"ID":"a82ed89d-c83d-4146-91c7-30dbd3cd1d66","Type":"ContainerStarted","Data":"cd7ca7be17956d263abde1889e36d0b6348d127cfc32e8c6b3faca78b9c85010"} Sep 19 10:56:15.374288 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:15.374254 2579 generic.go:356] "Generic (PLEG): container finished" podID="a82ed89d-c83d-4146-91c7-30dbd3cd1d66" containerID="ffd59e3e04b5a55655694965b329ea46b142ecd440547212e3fe3df935f4d2f9" exitCode=0 Sep 19 10:56:15.374729 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:15.374340 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs" event={"ID":"a82ed89d-c83d-4146-91c7-30dbd3cd1d66","Type":"ContainerDied","Data":"ffd59e3e04b5a55655694965b329ea46b142ecd440547212e3fe3df935f4d2f9"} Sep 19 10:56:16.523201 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:16.523176 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs" Sep 19 10:56:16.593403 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:16.593379 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82ed89d-c83d-4146-91c7-30dbd3cd1d66-maas-api-service-ca\") pod \"a82ed89d-c83d-4146-91c7-30dbd3cd1d66\" (UID: \"a82ed89d-c83d-4146-91c7-30dbd3cd1d66\") " Sep 19 10:56:16.593701 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:16.593680 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82ed89d-c83d-4146-91c7-30dbd3cd1d66-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a82ed89d-c83d-4146-91c7-30dbd3cd1d66" (UID: "a82ed89d-c83d-4146-91c7-30dbd3cd1d66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:56:16.694928 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:16.694900 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82ed89d-c83d-4146-91c7-30dbd3cd1d66-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:56:17.386969 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:17.386935 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs" Sep 19 10:56:17.387185 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:17.386934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs" event={"ID":"a82ed89d-c83d-4146-91c7-30dbd3cd1d66","Type":"ContainerDied","Data":"cd7ca7be17956d263abde1889e36d0b6348d127cfc32e8c6b3faca78b9c85010"} Sep 19 10:56:17.387185 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:17.387087 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd7ca7be17956d263abde1889e36d0b6348d127cfc32e8c6b3faca78b9c85010" Sep 19 10:56:26.695710 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:26.695675 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8"] Sep 19 10:56:26.696160 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:26.696146 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82ed89d-c83d-4146-91c7-30dbd3cd1d66" containerName="revoke-keys" Sep 19 10:56:26.696160 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:26.696161 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82ed89d-c83d-4146-91c7-30dbd3cd1d66" containerName="revoke-keys" Sep 19 10:56:26.696259 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:26.696222 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6939ddf4-8535-4764-ac63-12646e8a14bb" containerName="istio-proxy" Sep 19 10:56:26.696259 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:26.696233 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a82ed89d-c83d-4146-91c7-30dbd3cd1d66" containerName="revoke-keys" Sep 19 10:56:26.699255 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:26.699236 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8" Sep 19 10:56:26.701514 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:26.701491 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 10:56:26.701611 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:26.701534 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 10:56:26.705498 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:26.705477 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8"] Sep 19 10:56:26.791259 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:26.791231 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a006fb-b3a0-49a1-aa49-1f6f50a2027f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8\" (UID: \"f4a006fb-b3a0-49a1-aa49-1f6f50a2027f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8" Sep 19 10:56:26.892491 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:26.892458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a006fb-b3a0-49a1-aa49-1f6f50a2027f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8\" (UID: \"f4a006fb-b3a0-49a1-aa49-1f6f50a2027f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8" Sep 19 10:56:26.893083 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:26.893039 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a006fb-b3a0-49a1-aa49-1f6f50a2027f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8\" (UID: \"f4a006fb-b3a0-49a1-aa49-1f6f50a2027f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8" Sep 19 10:56:27.010428 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:27.010342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8" Sep 19 10:56:27.281761 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:27.281733 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8"] Sep 19 10:56:27.283658 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:56:27.283632 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a006fb_b3a0_49a1_aa49_1f6f50a2027f.slice/crio-1fd0b4fa64ef63d10bd989b8d1cd325c249ea72bd0e65d970b5ad27b38b5053f WatchSource:0}: Error finding container 1fd0b4fa64ef63d10bd989b8d1cd325c249ea72bd0e65d970b5ad27b38b5053f: Status 404 returned error can't find the container with id 1fd0b4fa64ef63d10bd989b8d1cd325c249ea72bd0e65d970b5ad27b38b5053f Sep 19 10:56:27.447820 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:27.447791 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8" event={"ID":"f4a006fb-b3a0-49a1-aa49-1f6f50a2027f","Type":"ContainerStarted","Data":"1fd0b4fa64ef63d10bd989b8d1cd325c249ea72bd0e65d970b5ad27b38b5053f"} Sep 19 10:56:28.454339 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:28.454301 2579 generic.go:356] "Generic (PLEG): container finished" podID="f4a006fb-b3a0-49a1-aa49-1f6f50a2027f" containerID="af60fc7cb2b3d93796eed5e735db6032a9d451b56d95645ecf25a01f416c09d7" exitCode=0 Sep 19 10:56:28.454729 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:28.454381 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8" event={"ID":"f4a006fb-b3a0-49a1-aa49-1f6f50a2027f","Type":"ContainerDied","Data":"af60fc7cb2b3d93796eed5e735db6032a9d451b56d95645ecf25a01f416c09d7"} Sep 19 10:56:29.031903 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:29.031867 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs"] Sep 19 10:56:29.033851 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:29.033830 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6f44882-7wzvs"] Sep 19 10:56:29.593030 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:29.593007 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8" Sep 19 10:56:29.726306 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:29.726224 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a006fb-b3a0-49a1-aa49-1f6f50a2027f-maas-api-service-ca\") pod \"f4a006fb-b3a0-49a1-aa49-1f6f50a2027f\" (UID: \"f4a006fb-b3a0-49a1-aa49-1f6f50a2027f\") " Sep 19 10:56:29.726560 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:29.726537 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a006fb-b3a0-49a1-aa49-1f6f50a2027f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4a006fb-b3a0-49a1-aa49-1f6f50a2027f" (UID: "f4a006fb-b3a0-49a1-aa49-1f6f50a2027f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:56:29.827984 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:29.827953 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a006fb-b3a0-49a1-aa49-1f6f50a2027f-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:56:30.199396 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:30.199359 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a82ed89d-c83d-4146-91c7-30dbd3cd1d66" path="/var/lib/kubelet/pods/a82ed89d-c83d-4146-91c7-30dbd3cd1d66/volumes" Sep 19 10:56:30.403722 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:30.403695 2579 scope.go:117] "RemoveContainer" containerID="44bd53cfd09158c76d50becf9f153380f3a5d1265a3f8e1441673145998b8a61" Sep 19 10:56:30.467081 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:30.467017 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8" Sep 19 10:56:30.467081 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:30.467054 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8" event={"ID":"f4a006fb-b3a0-49a1-aa49-1f6f50a2027f","Type":"ContainerDied","Data":"1fd0b4fa64ef63d10bd989b8d1cd325c249ea72bd0e65d970b5ad27b38b5053f"} Sep 19 10:56:30.467221 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:30.467108 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fd0b4fa64ef63d10bd989b8d1cd325c249ea72bd0e65d970b5ad27b38b5053f" Sep 19 10:56:42.615187 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:42.615020 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8"] Sep 19 10:56:42.617289 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:42.617263 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc1b19-kbkd8"] Sep 19 10:56:44.196631 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:56:44.196543 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4a006fb-b3a0-49a1-aa49-1f6f50a2027f" path="/var/lib/kubelet/pods/f4a006fb-b3a0-49a1-aa49-1f6f50a2027f/volumes" Sep 19 10:57:19.718506 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:19.718467 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk"] Sep 19 10:57:19.719083 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:19.718764 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" containerID="cri-o://f04bdaf803bdf2487febec117c2e3ade0ad9dc8a45a9a4a59c29a72273e1aacf" gracePeriod=30 Sep 19 10:57:24.980124 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:24.980103 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:57:25.055712 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.055645 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-envoy\") pod \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " Sep 19 10:57:25.055712 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.055682 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-workload-socket\") pod \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " Sep 19 10:57:25.055900 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.055715 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-podinfo\") pod \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " Sep 19 10:57:25.055900 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.055778 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istiod-ca-cert\") pod \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " Sep 19 10:57:25.055900 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.055797 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-workload-certs\") pod \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " Sep 19 10:57:25.055900 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.055834 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-data\") pod \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " Sep 19 10:57:25.055900 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.055869 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-credential-socket\") pod \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " Sep 19 10:57:25.055900 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.055896 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwrdr\" (UniqueName: \"kubernetes.io/projected/2b3a8639-373e-4d27-83c2-02eefc29d0cd-kube-api-access-mwrdr\") pod \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " Sep 19 10:57:25.056235 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.055954 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-token\") pod \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\" (UID: \"2b3a8639-373e-4d27-83c2-02eefc29d0cd\") " Sep 19 10:57:25.056235 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.055974 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2b3a8639-373e-4d27-83c2-02eefc29d0cd" (UID: "2b3a8639-373e-4d27-83c2-02eefc29d0cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:57:25.056235 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.056119 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2b3a8639-373e-4d27-83c2-02eefc29d0cd" (UID: "2b3a8639-373e-4d27-83c2-02eefc29d0cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:57:25.056235 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.056206 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2b3a8639-373e-4d27-83c2-02eefc29d0cd" (UID: "2b3a8639-373e-4d27-83c2-02eefc29d0cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:57:25.056451 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.056263 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2b3a8639-373e-4d27-83c2-02eefc29d0cd" (UID: "2b3a8639-373e-4d27-83c2-02eefc29d0cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:57:25.056451 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.056336 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-data" (OuterVolumeSpecName: "istio-data") pod "2b3a8639-373e-4d27-83c2-02eefc29d0cd" (UID: "2b3a8639-373e-4d27-83c2-02eefc29d0cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:57:25.056451 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.056439 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:57:25.056604 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.056457 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:57:25.056604 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.056471 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:57:25.056604 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.056483 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:57:25.056604 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.056495 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:57:25.057973 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.057944 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2b3a8639-373e-4d27-83c2-02eefc29d0cd" (UID: "2b3a8639-373e-4d27-83c2-02eefc29d0cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 10:57:25.058159 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.058134 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3a8639-373e-4d27-83c2-02eefc29d0cd-kube-api-access-mwrdr" (OuterVolumeSpecName: "kube-api-access-mwrdr") pod "2b3a8639-373e-4d27-83c2-02eefc29d0cd" (UID: "2b3a8639-373e-4d27-83c2-02eefc29d0cd"). InnerVolumeSpecName "kube-api-access-mwrdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:57:25.058254 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.058207 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2b3a8639-373e-4d27-83c2-02eefc29d0cd" (UID: "2b3a8639-373e-4d27-83c2-02eefc29d0cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 10:57:25.058254 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.058215 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-token" (OuterVolumeSpecName: "istio-token") pod "2b3a8639-373e-4d27-83c2-02eefc29d0cd" (UID: "2b3a8639-373e-4d27-83c2-02eefc29d0cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:57:25.157657 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.157630 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:57:25.157657 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.157652 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwrdr\" (UniqueName: \"kubernetes.io/projected/2b3a8639-373e-4d27-83c2-02eefc29d0cd-kube-api-access-mwrdr\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:57:25.157807 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.157663 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:57:25.157807 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.157671 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3a8639-373e-4d27-83c2-02eefc29d0cd-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:57:25.787121 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.787087 2579 generic.go:356] "Generic (PLEG): container finished" podID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerID="f04bdaf803bdf2487febec117c2e3ade0ad9dc8a45a9a4a59c29a72273e1aacf" exitCode=0 Sep 19 10:57:25.787289 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.787134 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" Sep 19 10:57:25.787289 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.787151 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" event={"ID":"2b3a8639-373e-4d27-83c2-02eefc29d0cd","Type":"ContainerDied","Data":"f04bdaf803bdf2487febec117c2e3ade0ad9dc8a45a9a4a59c29a72273e1aacf"} Sep 19 10:57:25.787289 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.787241 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk" event={"ID":"2b3a8639-373e-4d27-83c2-02eefc29d0cd","Type":"ContainerDied","Data":"9509e705aa87beca1155e78d8df7f8bb24847e0127eb64cab61f97b3c1d3bcb7"} Sep 19 10:57:25.787289 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.787271 2579 scope.go:117] "RemoveContainer" containerID="f04bdaf803bdf2487febec117c2e3ade0ad9dc8a45a9a4a59c29a72273e1aacf" Sep 19 10:57:25.797470 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.797453 2579 scope.go:117] "RemoveContainer" containerID="c4457ce61563ef951b3ecb847fcf059a7a1640b4198b6347739eac66da43045e" Sep 19 10:57:25.805740 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.805724 2579 scope.go:117] "RemoveContainer" containerID="f04bdaf803bdf2487febec117c2e3ade0ad9dc8a45a9a4a59c29a72273e1aacf" Sep 19 10:57:25.806012 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:57:25.805993 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04bdaf803bdf2487febec117c2e3ade0ad9dc8a45a9a4a59c29a72273e1aacf\": container with ID starting with f04bdaf803bdf2487febec117c2e3ade0ad9dc8a45a9a4a59c29a72273e1aacf not found: ID does not exist" containerID="f04bdaf803bdf2487febec117c2e3ade0ad9dc8a45a9a4a59c29a72273e1aacf" Sep 19 10:57:25.806083 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.806021 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04bdaf803bdf2487febec117c2e3ade0ad9dc8a45a9a4a59c29a72273e1aacf"} err="failed to get container status \"f04bdaf803bdf2487febec117c2e3ade0ad9dc8a45a9a4a59c29a72273e1aacf\": rpc error: code = NotFound desc = could not find container \"f04bdaf803bdf2487febec117c2e3ade0ad9dc8a45a9a4a59c29a72273e1aacf\": container with ID starting with f04bdaf803bdf2487febec117c2e3ade0ad9dc8a45a9a4a59c29a72273e1aacf not found: ID does not exist" Sep 19 10:57:25.806083 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.806038 2579 scope.go:117] "RemoveContainer" containerID="c4457ce61563ef951b3ecb847fcf059a7a1640b4198b6347739eac66da43045e" Sep 19 10:57:25.806435 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:57:25.806407 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4457ce61563ef951b3ecb847fcf059a7a1640b4198b6347739eac66da43045e\": container with ID starting with c4457ce61563ef951b3ecb847fcf059a7a1640b4198b6347739eac66da43045e not found: ID does not exist" containerID="c4457ce61563ef951b3ecb847fcf059a7a1640b4198b6347739eac66da43045e" Sep 19 10:57:25.806586 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.806441 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4457ce61563ef951b3ecb847fcf059a7a1640b4198b6347739eac66da43045e"} err="failed to get container status \"c4457ce61563ef951b3ecb847fcf059a7a1640b4198b6347739eac66da43045e\": rpc error: code = NotFound desc = could not find container \"c4457ce61563ef951b3ecb847fcf059a7a1640b4198b6347739eac66da43045e\": container with ID starting with c4457ce61563ef951b3ecb847fcf059a7a1640b4198b6347739eac66da43045e not found: ID does not exist" Sep 19 10:57:25.809217 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.809198 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk"] Sep 19 10:57:25.812655 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:25.812635 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk"] Sep 19 10:57:26.196638 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:57:26.196609 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" path="/var/lib/kubelet/pods/2b3a8639-373e-4d27-83c2-02eefc29d0cd/volumes" Sep 19 10:58:20.165992 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.165956 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx"] Sep 19 10:58:20.166447 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.166425 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a006fb-b3a0-49a1-aa49-1f6f50a2027f" containerName="revoke-keys" Sep 19 10:58:20.166447 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.166438 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a006fb-b3a0-49a1-aa49-1f6f50a2027f" containerName="revoke-keys" Sep 19 10:58:20.166552 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.166455 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" Sep 19 10:58:20.166552 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.166461 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" Sep 19 10:58:20.166552 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.166547 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" Sep 19 10:58:20.166660 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.166558 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" Sep 19 10:58:20.166660 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.166564 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a006fb-b3a0-49a1-aa49-1f6f50a2027f" containerName="revoke-keys" Sep 19 10:58:20.169734 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.169718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" Sep 19 10:58:20.172299 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.172274 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-45f425-dockercfg-89flh\"" Sep 19 10:58:20.172299 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.172295 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-45f425\"" Sep 19 10:58:20.180183 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.180160 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx"] Sep 19 10:58:20.262826 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.262779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx\" (UID: \"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" Sep 19 10:58:20.263018 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.262888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ljxk\" (UniqueName: \"kubernetes.io/projected/544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64-kube-api-access-4ljxk\") pod \"payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx\" (UID: \"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" Sep 19 10:58:20.364196 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.364163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx\" (UID: \"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" Sep 19 10:58:20.364381 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.364228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ljxk\" (UniqueName: \"kubernetes.io/projected/544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64-kube-api-access-4ljxk\") pod \"payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx\" (UID: \"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" Sep 19 10:58:20.364889 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.364862 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx\" (UID: \"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" Sep 19 10:58:20.373041 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.373017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ljxk\" (UniqueName: \"kubernetes.io/projected/544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64-kube-api-access-4ljxk\") pod \"payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx\" (UID: \"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" Sep 19 10:58:20.480380 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.480313 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" Sep 19 10:58:20.755216 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:20.755189 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx"] Sep 19 10:58:20.757325 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:58:20.757290 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544fc7e3_f4ea_44c2_a6df_cb1c2ad8fd64.slice/crio-2088635de963d62bfc8673e59f204a809e3ec496f1d511a3d051ed2257953956 WatchSource:0}: Error finding container 2088635de963d62bfc8673e59f204a809e3ec496f1d511a3d051ed2257953956: Status 404 returned error can't find the container with id 2088635de963d62bfc8673e59f204a809e3ec496f1d511a3d051ed2257953956 Sep 19 10:58:21.099550 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:21.099450 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" event={"ID":"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64","Type":"ContainerStarted","Data":"522df49c69f53a836823dd386a5474b5e4c51d943be0aa9a404d3bca0040d279"} Sep 19 10:58:21.099698 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:21.099614 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" Sep 19 10:58:21.099698 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:21.099652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" event={"ID":"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64","Type":"ContainerStarted","Data":"2088635de963d62bfc8673e59f204a809e3ec496f1d511a3d051ed2257953956"} Sep 19 10:58:21.116110 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:21.116040 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" podStartSLOduration=1.116025119 podStartE2EDuration="1.116025119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:58:21.113733131 +0000 UTC m=+1433.026237847" watchObservedRunningTime="2026-09-19 10:58:21.116025119 +0000 UTC m=+1433.028529836" Sep 19 10:58:32.107237 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:32.107202 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" Sep 19 10:58:47.439031 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.438995 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq"] Sep 19 10:58:47.439760 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.439735 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" Sep 19 10:58:47.440014 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.439985 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3a8639-373e-4d27-83c2-02eefc29d0cd" containerName="istio-proxy" Sep 19 10:58:47.444158 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.444135 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.446562 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.446541 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a1c584-openshift-default-dockercfg-45q95\"" Sep 19 10:58:47.453318 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.453255 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq"] Sep 19 10:58:47.532513 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.532223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-envoy\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.532513 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.532287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-token\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.532513 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.532336 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-workload-certs\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.532513 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.532360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975e4e10-588f-4eeb-af95-ae61197f97e9-istiod-ca-cert\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.532513 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.532393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-workload-socket\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.532513 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.532422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-podinfo\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.532513 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.532470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-data\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.532513 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.532497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-credential-socket\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.533141 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.532621 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjp2\" (UniqueName: \"kubernetes.io/projected/975e4e10-588f-4eeb-af95-ae61197f97e9-kube-api-access-8kjp2\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.634141 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.634104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjp2\" (UniqueName: \"kubernetes.io/projected/975e4e10-588f-4eeb-af95-ae61197f97e9-kube-api-access-8kjp2\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.634338 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.634183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-envoy\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.634338 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.634222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-token\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.634338 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.634265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-workload-certs\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.634338 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.634286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975e4e10-588f-4eeb-af95-ae61197f97e9-istiod-ca-cert\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.634338 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.634316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-workload-socket\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.634615 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.634341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-podinfo\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.634615 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.634388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-data\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.634615 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.634416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-credential-socket\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.634962 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.634930 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-credential-socket\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.636268 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.636240 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975e4e10-588f-4eeb-af95-ae61197f97e9-istiod-ca-cert\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.636374 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.636281 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-data\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.636445 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.636428 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-workload-socket\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.636674 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.636650 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-workload-certs\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.638968 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.638942 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-envoy\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.639349 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.639327 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-podinfo\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.643675 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.643653 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjp2\" (UniqueName: \"kubernetes.io/projected/975e4e10-588f-4eeb-af95-ae61197f97e9-kube-api-access-8kjp2\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.643778 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.643689 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-token\") pod \"e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:47.757945 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:47.757867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:48.033881 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:48.033830 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq"] Sep 19 10:58:48.036183 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:58:48.036160 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975e4e10_588f_4eeb_af95_ae61197f97e9.slice/crio-aa593c365b7d42ea3bf478d4e4771312f7f21c940619a3b3b70293753fb6f852 WatchSource:0}: Error finding container aa593c365b7d42ea3bf478d4e4771312f7f21c940619a3b3b70293753fb6f852: Status 404 returned error can't find the container with id aa593c365b7d42ea3bf478d4e4771312f7f21c940619a3b3b70293753fb6f852 Sep 19 10:58:48.038494 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:48.038461 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:58:48.038592 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:48.038539 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:58:48.038666 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:48.038621 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:58:48.260446 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:48.260379 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" event={"ID":"975e4e10-588f-4eeb-af95-ae61197f97e9","Type":"ContainerStarted","Data":"31aba3ddf08c1ed34aabcce57d3495a299666d63416336888157b5cd20791d3f"} Sep 19 10:58:48.260446 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:48.260434 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" event={"ID":"975e4e10-588f-4eeb-af95-ae61197f97e9","Type":"ContainerStarted","Data":"aa593c365b7d42ea3bf478d4e4771312f7f21c940619a3b3b70293753fb6f852"} Sep 19 10:58:48.286244 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:48.286133 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" podStartSLOduration=1.28611337 podStartE2EDuration="1.28611337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:58:48.28497392 +0000 UTC m=+1460.197478637" watchObservedRunningTime="2026-09-19 10:58:48.28611337 +0000 UTC m=+1460.198618089" Sep 19 10:58:48.758542 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:48.758510 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:49.763691 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:49.763662 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:50.271841 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:50.271808 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:58:50.272853 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:58:50.272836 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:59:11.401135 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:11.401101 2579 generic.go:356] "Generic (PLEG): container finished" podID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerID="31aba3ddf08c1ed34aabcce57d3495a299666d63416336888157b5cd20791d3f" exitCode=0 Sep 19 10:59:11.401515 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:11.401175 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" event={"ID":"975e4e10-588f-4eeb-af95-ae61197f97e9","Type":"ContainerDied","Data":"31aba3ddf08c1ed34aabcce57d3495a299666d63416336888157b5cd20791d3f"} Sep 19 10:59:11.401581 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:11.401567 2579 scope.go:117] "RemoveContainer" containerID="31aba3ddf08c1ed34aabcce57d3495a299666d63416336888157b5cd20791d3f" Sep 19 10:59:11.402712 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:11.402664 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:59:11.402788 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:11.402756 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:59:11.402825 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:11.402786 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 19 10:59:12.408902 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:12.408870 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" event={"ID":"975e4e10-588f-4eeb-af95-ae61197f97e9","Type":"ContainerStarted","Data":"672134461a305a7ca3b65556ef92d0a4d9a761db8e25c5d44c374e503ef659cb"} Sep 19 10:59:12.759744 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:12.759101 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:59:12.759744 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:12.759358 2579 patch_prober.go:28] interesting pod/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 19 10:59:12.759744 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:12.759429 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" podUID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 19 10:59:13.759206 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:13.759168 2579 patch_prober.go:28] interesting pod/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 19 10:59:13.759697 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:13.759237 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" podUID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 19 10:59:14.758569 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:14.758525 2579 patch_prober.go:28] interesting pod/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 19 10:59:14.758794 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:14.758593 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" podUID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 19 10:59:15.759279 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:15.759241 2579 patch_prober.go:28] interesting pod/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 19 10:59:15.759770 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:15.759311 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" podUID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 19 10:59:16.759078 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:16.759026 2579 patch_prober.go:28] interesting pod/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 19 10:59:16.759298 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:16.759117 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" podUID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 19 10:59:17.909434 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:17.909388 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:59:17.910031 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:17.910006 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:59:17.911490 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:17.911466 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 10:59:23.073974 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:23.073941 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt"] Sep 19 10:59:23.076767 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:23.076750 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt" Sep 19 10:59:23.079030 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:23.079008 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 10:59:23.079149 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:23.079102 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 10:59:23.084401 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:23.084378 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt"] Sep 19 10:59:23.188505 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:23.188467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c3ee97-c439-4641-b4ff-a91d5c4e6c21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-45f425-lhvpt\" (UID: \"76c3ee97-c439-4641-b4ff-a91d5c4e6c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt" Sep 19 10:59:23.289298 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:23.289264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c3ee97-c439-4641-b4ff-a91d5c4e6c21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-45f425-lhvpt\" (UID: \"76c3ee97-c439-4641-b4ff-a91d5c4e6c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt" Sep 19 10:59:23.289851 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:23.289834 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c3ee97-c439-4641-b4ff-a91d5c4e6c21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-45f425-lhvpt\" (UID: \"76c3ee97-c439-4641-b4ff-a91d5c4e6c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt" Sep 19 10:59:23.389503 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:23.389472 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt" Sep 19 10:59:23.662766 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:23.662742 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt"] Sep 19 10:59:23.664592 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:59:23.664564 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c3ee97_c439_4641_b4ff_a91d5c4e6c21.slice/crio-8a40e63f80a0876009c61b42654f386674252292528a4e338f621276c763212c WatchSource:0}: Error finding container 8a40e63f80a0876009c61b42654f386674252292528a4e338f621276c763212c: Status 404 returned error can't find the container with id 8a40e63f80a0876009c61b42654f386674252292528a4e338f621276c763212c Sep 19 10:59:24.479858 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:24.479820 2579 generic.go:356] "Generic (PLEG): container finished" podID="76c3ee97-c439-4641-b4ff-a91d5c4e6c21" containerID="0190e33bacf489c92ecacaab15a27343853ff2960d4e80164fb4f4933f68078a" exitCode=0 Sep 19 10:59:24.480254 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:24.479905 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt" event={"ID":"76c3ee97-c439-4641-b4ff-a91d5c4e6c21","Type":"ContainerDied","Data":"0190e33bacf489c92ecacaab15a27343853ff2960d4e80164fb4f4933f68078a"} Sep 19 10:59:24.480254 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:24.479960 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt" event={"ID":"76c3ee97-c439-4641-b4ff-a91d5c4e6c21","Type":"ContainerStarted","Data":"8a40e63f80a0876009c61b42654f386674252292528a4e338f621276c763212c"} Sep 19 10:59:25.625492 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:25.625468 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt" Sep 19 10:59:25.713687 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:25.713655 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c3ee97-c439-4641-b4ff-a91d5c4e6c21-maas-api-service-ca\") pod \"76c3ee97-c439-4641-b4ff-a91d5c4e6c21\" (UID: \"76c3ee97-c439-4641-b4ff-a91d5c4e6c21\") " Sep 19 10:59:25.714004 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:25.713981 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c3ee97-c439-4641-b4ff-a91d5c4e6c21-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76c3ee97-c439-4641-b4ff-a91d5c4e6c21" (UID: "76c3ee97-c439-4641-b4ff-a91d5c4e6c21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:59:25.815338 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:25.815264 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c3ee97-c439-4641-b4ff-a91d5c4e6c21-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:59:26.493052 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:26.492981 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt" Sep 19 10:59:26.493052 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:26.493018 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt" event={"ID":"76c3ee97-c439-4641-b4ff-a91d5c4e6c21","Type":"ContainerDied","Data":"8a40e63f80a0876009c61b42654f386674252292528a4e338f621276c763212c"} Sep 19 10:59:26.493247 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:26.493098 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a40e63f80a0876009c61b42654f386674252292528a4e338f621276c763212c" Sep 19 10:59:30.292967 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:30.292940 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 10:59:30.294517 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:30.294493 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 10:59:30.462482 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:30.462457 2579 scope.go:117] "RemoveContainer" containerID="36430b7b10657a3efee431b47a7ee8239cee26a0e6bb6bfb5c05087f799e7e06" Sep 19 10:59:30.471006 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:30.470986 2579 scope.go:117] "RemoveContainer" containerID="4f243e3969fedd3474e16be408430bd3d7d818dca9e70e2751b5f7c1a3361039" Sep 19 10:59:30.478285 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:30.478269 2579 scope.go:117] "RemoveContainer" containerID="dc46d29359936df1cea34b5eaf19453ef50a469be844a2cb92aa137585270672" Sep 19 10:59:38.579291 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:38.579245 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx"] Sep 19 10:59:38.579752 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:38.579477 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" podUID="544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64" containerName="payload-pre-processing" containerID="cri-o://522df49c69f53a836823dd386a5474b5e4c51d943be0aa9a404d3bca0040d279" gracePeriod=30 Sep 19 10:59:38.724805 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:38.724723 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt"] Sep 19 10:59:38.727045 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:38.727018 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45f425-lhvpt"] Sep 19 10:59:38.839108 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:38.839036 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" Sep 19 10:59:38.940506 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:38.940463 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64-plugins-config-volume\") pod \"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64\" (UID: \"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64\") " Sep 19 10:59:38.940713 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:38.940519 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ljxk\" (UniqueName: \"kubernetes.io/projected/544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64-kube-api-access-4ljxk\") pod \"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64\" (UID: \"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64\") " Sep 19 10:59:38.940941 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:38.940910 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64" (UID: "544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:59:38.942766 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:38.942733 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64-kube-api-access-4ljxk" (OuterVolumeSpecName: "kube-api-access-4ljxk") pod "544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64" (UID: "544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64"). InnerVolumeSpecName "kube-api-access-4ljxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 10:59:39.042179 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:39.042147 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:59:39.042179 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:39.042172 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ljxk\" (UniqueName: \"kubernetes.io/projected/544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64-kube-api-access-4ljxk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:59:39.566007 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:39.565969 2579 generic.go:356] "Generic (PLEG): container finished" podID="544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64" containerID="522df49c69f53a836823dd386a5474b5e4c51d943be0aa9a404d3bca0040d279" exitCode=0 Sep 19 10:59:39.566308 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:39.566016 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" Sep 19 10:59:39.566308 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:39.566076 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" event={"ID":"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64","Type":"ContainerDied","Data":"522df49c69f53a836823dd386a5474b5e4c51d943be0aa9a404d3bca0040d279"} Sep 19 10:59:39.566308 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:39.566136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx" event={"ID":"544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64","Type":"ContainerDied","Data":"2088635de963d62bfc8673e59f204a809e3ec496f1d511a3d051ed2257953956"} Sep 19 10:59:39.566308 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:39.566159 2579 scope.go:117] "RemoveContainer" containerID="522df49c69f53a836823dd386a5474b5e4c51d943be0aa9a404d3bca0040d279" Sep 19 10:59:39.575791 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:39.575781 2579 scope.go:117] "RemoveContainer" containerID="522df49c69f53a836823dd386a5474b5e4c51d943be0aa9a404d3bca0040d279" Sep 19 10:59:39.576085 ip-10-0-143-38 kubenswrapper[2579]: E0919 10:59:39.576045 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522df49c69f53a836823dd386a5474b5e4c51d943be0aa9a404d3bca0040d279\": container with ID starting with 522df49c69f53a836823dd386a5474b5e4c51d943be0aa9a404d3bca0040d279 not found: ID does not exist" containerID="522df49c69f53a836823dd386a5474b5e4c51d943be0aa9a404d3bca0040d279" Sep 19 10:59:39.576179 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:39.576086 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522df49c69f53a836823dd386a5474b5e4c51d943be0aa9a404d3bca0040d279"} err="failed to get container status \"522df49c69f53a836823dd386a5474b5e4c51d943be0aa9a404d3bca0040d279\": rpc error: code = NotFound desc = could not find container \"522df49c69f53a836823dd386a5474b5e4c51d943be0aa9a404d3bca0040d279\": container with ID starting with 522df49c69f53a836823dd386a5474b5e4c51d943be0aa9a404d3bca0040d279 not found: ID does not exist" Sep 19 10:59:39.587695 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:39.587672 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx"] Sep 19 10:59:39.589693 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:39.589673 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx"] Sep 19 10:59:40.198369 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:40.198337 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64" path="/var/lib/kubelet/pods/544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64/volumes" Sep 19 10:59:40.198703 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:40.198689 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76c3ee97-c439-4641-b4ff-a91d5c4e6c21" path="/var/lib/kubelet/pods/76c3ee97-c439-4641-b4ff-a91d5c4e6c21/volumes" Sep 19 10:59:45.513504 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.513429 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2"] Sep 19 10:59:45.513889 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.513875 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c3ee97-c439-4641-b4ff-a91d5c4e6c21" containerName="revoke-keys" Sep 19 10:59:45.513889 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.513889 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c3ee97-c439-4641-b4ff-a91d5c4e6c21" containerName="revoke-keys" Sep 19 10:59:45.513969 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.513908 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64" containerName="payload-pre-processing" Sep 19 10:59:45.513969 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.513914 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64" containerName="payload-pre-processing" Sep 19 10:59:45.514040 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.513975 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="544fc7e3-f4ea-44c2-a6df-cb1c2ad8fd64" containerName="payload-pre-processing" Sep 19 10:59:45.514040 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.513984 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c3ee97-c439-4641-b4ff-a91d5c4e6c21" containerName="revoke-keys" Sep 19 10:59:45.518536 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.518519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2" Sep 19 10:59:45.521383 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.521361 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 10:59:45.521532 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.521409 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 10:59:45.523310 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.523284 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2"] Sep 19 10:59:45.606401 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.606372 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e22fdce-83c6-412c-8a74-620c6e70c15b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a1c584-v29t2\" (UID: \"8e22fdce-83c6-412c-8a74-620c6e70c15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2" Sep 19 10:59:45.707819 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.707788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e22fdce-83c6-412c-8a74-620c6e70c15b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a1c584-v29t2\" (UID: \"8e22fdce-83c6-412c-8a74-620c6e70c15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2" Sep 19 10:59:45.708432 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.708411 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e22fdce-83c6-412c-8a74-620c6e70c15b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a1c584-v29t2\" (UID: \"8e22fdce-83c6-412c-8a74-620c6e70c15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2" Sep 19 10:59:45.830889 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:45.830804 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2" Sep 19 10:59:46.108461 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:46.108379 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2"] Sep 19 10:59:46.111512 ip-10-0-143-38 kubenswrapper[2579]: W0919 10:59:46.111478 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e22fdce_83c6_412c_8a74_620c6e70c15b.slice/crio-69bdd49af17a1bc6b188cf429f54d53bdedf7525ff91b069b3244a4b35196343 WatchSource:0}: Error finding container 69bdd49af17a1bc6b188cf429f54d53bdedf7525ff91b069b3244a4b35196343: Status 404 returned error can't find the container with id 69bdd49af17a1bc6b188cf429f54d53bdedf7525ff91b069b3244a4b35196343 Sep 19 10:59:46.610819 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:46.610760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2" event={"ID":"8e22fdce-83c6-412c-8a74-620c6e70c15b","Type":"ContainerStarted","Data":"99262bd826c8842746e4fddfdd3fa825f045473f97474167686824b2f349fa24"} Sep 19 10:59:46.611220 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:46.610850 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2" event={"ID":"8e22fdce-83c6-412c-8a74-620c6e70c15b","Type":"ContainerStarted","Data":"69bdd49af17a1bc6b188cf429f54d53bdedf7525ff91b069b3244a4b35196343"} Sep 19 10:59:46.631791 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:46.631699 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2" podStartSLOduration=1.631683357 podStartE2EDuration="1.631683357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 10:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 10:59:46.62997872 +0000 UTC m=+1518.542483640" watchObservedRunningTime="2026-09-19 10:59:46.631683357 +0000 UTC m=+1518.544188073" Sep 19 10:59:47.626115 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:47.626036 2579 generic.go:356] "Generic (PLEG): container finished" podID="8e22fdce-83c6-412c-8a74-620c6e70c15b" containerID="99262bd826c8842746e4fddfdd3fa825f045473f97474167686824b2f349fa24" exitCode=0 Sep 19 10:59:47.626478 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:47.626122 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2" event={"ID":"8e22fdce-83c6-412c-8a74-620c6e70c15b","Type":"ContainerDied","Data":"99262bd826c8842746e4fddfdd3fa825f045473f97474167686824b2f349fa24"} Sep 19 10:59:48.773535 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:48.773511 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2" Sep 19 10:59:48.938569 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:48.938537 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e22fdce-83c6-412c-8a74-620c6e70c15b-maas-api-service-ca\") pod \"8e22fdce-83c6-412c-8a74-620c6e70c15b\" (UID: \"8e22fdce-83c6-412c-8a74-620c6e70c15b\") " Sep 19 10:59:48.938920 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:48.938897 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e22fdce-83c6-412c-8a74-620c6e70c15b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e22fdce-83c6-412c-8a74-620c6e70c15b" (UID: "8e22fdce-83c6-412c-8a74-620c6e70c15b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 10:59:49.039653 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:49.039620 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e22fdce-83c6-412c-8a74-620c6e70c15b-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 10:59:49.638418 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:49.638387 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2" Sep 19 10:59:49.638655 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:49.638422 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2" event={"ID":"8e22fdce-83c6-412c-8a74-620c6e70c15b","Type":"ContainerDied","Data":"69bdd49af17a1bc6b188cf429f54d53bdedf7525ff91b069b3244a4b35196343"} Sep 19 10:59:49.638655 ip-10-0-143-38 kubenswrapper[2579]: I0919 10:59:49.638470 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69bdd49af17a1bc6b188cf429f54d53bdedf7525ff91b069b3244a4b35196343" Sep 19 11:00:00.136279 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.136243 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t"] Sep 19 11:00:00.136686 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.136672 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e22fdce-83c6-412c-8a74-620c6e70c15b" containerName="revoke-keys" Sep 19 11:00:00.136686 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.136686 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e22fdce-83c6-412c-8a74-620c6e70c15b" containerName="revoke-keys" Sep 19 11:00:00.136783 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.136763 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e22fdce-83c6-412c-8a74-620c6e70c15b" containerName="revoke-keys" Sep 19 11:00:00.139834 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.139815 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29830260-ldkmf"] Sep 19 11:00:00.139969 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.139947 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t" Sep 19 11:00:00.142730 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.142495 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 11:00:00.144113 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.144087 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8"] Sep 19 11:00:00.144746 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.144724 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-ldkmf" Sep 19 11:00:00.148126 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.148107 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn"] Sep 19 11:00:00.148347 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.148326 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8" Sep 19 11:00:00.152321 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.152296 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t"] Sep 19 11:00:00.152419 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.152334 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8"] Sep 19 11:00:00.152419 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.152409 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn" Sep 19 11:00:00.153719 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.153697 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn"] Sep 19 11:00:00.157001 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.156936 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t" Sep 19 11:00:00.172669 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.172467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-ldkmf" Sep 19 11:00:00.176238 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.175574 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29830260-ldkmf"] Sep 19 11:00:00.211713 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.211683 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8" Sep 19 11:00:00.219220 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.219073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn" Sep 19 11:00:00.456813 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.456787 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t"] Sep 19 11:00:00.458789 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:00:00.458765 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf786a48_3f02_4fe9_9915_082985c38b39.slice/crio-db9d4d8b95ff0406ebcf93516251c7b414fec267fc8cfa182e23ac3262d5af42 WatchSource:0}: Error finding container db9d4d8b95ff0406ebcf93516251c7b414fec267fc8cfa182e23ac3262d5af42: Status 404 returned error can't find the container with id db9d4d8b95ff0406ebcf93516251c7b414fec267fc8cfa182e23ac3262d5af42 Sep 19 11:00:00.466796 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.466760 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29830260-ldkmf"] Sep 19 11:00:00.468575 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:00:00.468554 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85c29cc_2f3c_4ff4_968b_e9d86275eed8.slice/crio-c45987fee24f4cf61019306d9a4199ea3899a1f5df5cd8e6bc2e42f9b1ea8e97 WatchSource:0}: Error finding container c45987fee24f4cf61019306d9a4199ea3899a1f5df5cd8e6bc2e42f9b1ea8e97: Status 404 returned error can't find the container with id c45987fee24f4cf61019306d9a4199ea3899a1f5df5cd8e6bc2e42f9b1ea8e97 Sep 19 11:00:00.522751 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.522715 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8"] Sep 19 11:00:00.524655 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:00:00.524627 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf09fd6b_6881_4c7f_82c4_b39046661850.slice/crio-674c6ca7ade5d54328e9c30eca2878ef45ffc28aa614bac43d5992adb4d4a248 WatchSource:0}: Error finding container 674c6ca7ade5d54328e9c30eca2878ef45ffc28aa614bac43d5992adb4d4a248: Status 404 returned error can't find the container with id 674c6ca7ade5d54328e9c30eca2878ef45ffc28aa614bac43d5992adb4d4a248 Sep 19 11:00:00.524984 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.524960 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn"] Sep 19 11:00:00.525672 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:00:00.525646 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e12cde_d509_47b2_a6c5_f1e25b9db808.slice/crio-c40c09d6fac49a9fce71a4a6da846c824bccf886c7ea48643ef3f5d4b273bb7f WatchSource:0}: Error finding container c40c09d6fac49a9fce71a4a6da846c824bccf886c7ea48643ef3f5d4b273bb7f: Status 404 returned error can't find the container with id c40c09d6fac49a9fce71a4a6da846c824bccf886c7ea48643ef3f5d4b273bb7f Sep 19 11:00:00.703614 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.703586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-ldkmf" event={"ID":"c85c29cc-2f3c-4ff4-968b-e9d86275eed8","Type":"ContainerStarted","Data":"c45987fee24f4cf61019306d9a4199ea3899a1f5df5cd8e6bc2e42f9b1ea8e97"} Sep 19 11:00:00.704795 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.704769 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8" event={"ID":"df09fd6b-6881-4c7f-82c4-b39046661850","Type":"ContainerStarted","Data":"674c6ca7ade5d54328e9c30eca2878ef45ffc28aa614bac43d5992adb4d4a248"} Sep 19 11:00:00.705952 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.705894 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t" event={"ID":"cf786a48-3f02-4fe9-9915-082985c38b39","Type":"ContainerStarted","Data":"db9d4d8b95ff0406ebcf93516251c7b414fec267fc8cfa182e23ac3262d5af42"} Sep 19 11:00:00.706996 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:00.706979 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn" event={"ID":"53e12cde-d509-47b2-a6c5-f1e25b9db808","Type":"ContainerStarted","Data":"c40c09d6fac49a9fce71a4a6da846c824bccf886c7ea48643ef3f5d4b273bb7f"} Sep 19 11:00:01.245349 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:01.245265 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2"] Sep 19 11:00:01.247219 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:01.247190 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a1c584-v29t2"] Sep 19 11:00:01.714575 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:01.714534 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-ldkmf" event={"ID":"c85c29cc-2f3c-4ff4-968b-e9d86275eed8","Type":"ContainerStarted","Data":"466bb2580938e800d764f38358564b15c72983edf6a940e7ba0d3207294a1a4c"} Sep 19 11:00:01.716111 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:01.716085 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8" event={"ID":"df09fd6b-6881-4c7f-82c4-b39046661850","Type":"ContainerStarted","Data":"8c639e7d34862b726b9e6a441419d291680d08b814a09c760724b29a054692f5"} Sep 19 11:00:01.717747 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:01.717726 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t" event={"ID":"cf786a48-3f02-4fe9-9915-082985c38b39","Type":"ContainerStarted","Data":"af05e0993154e1bc6e3001d78c42b035997d50d578f4fffa3f9986abeb702e1d"} Sep 19 11:00:01.719215 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:01.719194 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn" event={"ID":"53e12cde-d509-47b2-a6c5-f1e25b9db808","Type":"ContainerStarted","Data":"26ef5059518242a93d2cb8c72631e9f2801456ec52dd9c0000c60f1365e135c0"} Sep 19 11:00:01.730266 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:01.730228 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-ldkmf" podStartSLOduration=1.7302143509999999 podStartE2EDuration="1.730214351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:00:01.72915063 +0000 UTC m=+1533.641655347" watchObservedRunningTime="2026-09-19 11:00:01.730214351 +0000 UTC m=+1533.642719066" Sep 19 11:00:01.746355 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:01.746273 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t" podStartSLOduration=1.7462570290000001 podStartE2EDuration="1.746257029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:00:01.742730595 +0000 UTC m=+1533.655235315" watchObservedRunningTime="2026-09-19 11:00:01.746257029 +0000 UTC m=+1533.658761747" Sep 19 11:00:01.774174 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:01.774112 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn" podStartSLOduration=1.774091573 podStartE2EDuration="1.774091573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:00:01.759479959 +0000 UTC m=+1533.671984676" watchObservedRunningTime="2026-09-19 11:00:01.774091573 +0000 UTC m=+1533.686596281" Sep 19 11:00:01.774913 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:01.774873 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8" podStartSLOduration=1.774861925 podStartE2EDuration="1.774861925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:00:01.772375855 +0000 UTC m=+1533.684880577" watchObservedRunningTime="2026-09-19 11:00:01.774861925 +0000 UTC m=+1533.687366642" Sep 19 11:00:02.196530 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:02.196498 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e22fdce-83c6-412c-8a74-620c6e70c15b" path="/var/lib/kubelet/pods/8e22fdce-83c6-412c-8a74-620c6e70c15b/volumes" Sep 19 11:00:02.728134 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:02.728098 2579 generic.go:356] "Generic (PLEG): container finished" podID="53e12cde-d509-47b2-a6c5-f1e25b9db808" containerID="26ef5059518242a93d2cb8c72631e9f2801456ec52dd9c0000c60f1365e135c0" exitCode=0 Sep 19 11:00:02.728579 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:02.728191 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn" event={"ID":"53e12cde-d509-47b2-a6c5-f1e25b9db808","Type":"ContainerDied","Data":"26ef5059518242a93d2cb8c72631e9f2801456ec52dd9c0000c60f1365e135c0"} Sep 19 11:00:02.729957 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:02.729936 2579 generic.go:356] "Generic (PLEG): container finished" podID="c85c29cc-2f3c-4ff4-968b-e9d86275eed8" containerID="466bb2580938e800d764f38358564b15c72983edf6a940e7ba0d3207294a1a4c" exitCode=0 Sep 19 11:00:02.730106 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:02.730018 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-ldkmf" event={"ID":"c85c29cc-2f3c-4ff4-968b-e9d86275eed8","Type":"ContainerDied","Data":"466bb2580938e800d764f38358564b15c72983edf6a940e7ba0d3207294a1a4c"} Sep 19 11:00:02.731698 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:02.731671 2579 generic.go:356] "Generic (PLEG): container finished" podID="df09fd6b-6881-4c7f-82c4-b39046661850" containerID="8c639e7d34862b726b9e6a441419d291680d08b814a09c760724b29a054692f5" exitCode=0 Sep 19 11:00:02.731793 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:02.731745 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8" event={"ID":"df09fd6b-6881-4c7f-82c4-b39046661850","Type":"ContainerDied","Data":"8c639e7d34862b726b9e6a441419d291680d08b814a09c760724b29a054692f5"} Sep 19 11:00:02.733204 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:02.733187 2579 generic.go:356] "Generic (PLEG): container finished" podID="cf786a48-3f02-4fe9-9915-082985c38b39" containerID="af05e0993154e1bc6e3001d78c42b035997d50d578f4fffa3f9986abeb702e1d" exitCode=0 Sep 19 11:00:02.733289 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:02.733212 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t" event={"ID":"cf786a48-3f02-4fe9-9915-082985c38b39","Type":"ContainerDied","Data":"af05e0993154e1bc6e3001d78c42b035997d50d578f4fffa3f9986abeb702e1d"} Sep 19 11:00:03.914696 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:03.914667 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn" Sep 19 11:00:03.936044 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:03.935985 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-ldkmf" Sep 19 11:00:03.993092 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:03.993005 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t" Sep 19 11:00:03.997473 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:03.997455 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8" Sep 19 11:00:04.746638 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:04.746578 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8" Sep 19 11:00:04.746805 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:04.746578 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8" event={"ID":"df09fd6b-6881-4c7f-82c4-b39046661850","Type":"ContainerDied","Data":"674c6ca7ade5d54328e9c30eca2878ef45ffc28aa614bac43d5992adb4d4a248"} Sep 19 11:00:04.746805 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:04.746719 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="674c6ca7ade5d54328e9c30eca2878ef45ffc28aa614bac43d5992adb4d4a248" Sep 19 11:00:04.748115 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:04.748088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t" event={"ID":"cf786a48-3f02-4fe9-9915-082985c38b39","Type":"ContainerDied","Data":"db9d4d8b95ff0406ebcf93516251c7b414fec267fc8cfa182e23ac3262d5af42"} Sep 19 11:00:04.748223 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:04.748143 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db9d4d8b95ff0406ebcf93516251c7b414fec267fc8cfa182e23ac3262d5af42" Sep 19 11:00:04.748223 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:04.748096 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t" Sep 19 11:00:04.749423 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:04.749408 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn" Sep 19 11:00:04.749509 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:04.749441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn" event={"ID":"53e12cde-d509-47b2-a6c5-f1e25b9db808","Type":"ContainerDied","Data":"c40c09d6fac49a9fce71a4a6da846c824bccf886c7ea48643ef3f5d4b273bb7f"} Sep 19 11:00:04.749509 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:04.749475 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c40c09d6fac49a9fce71a4a6da846c824bccf886c7ea48643ef3f5d4b273bb7f" Sep 19 11:00:04.750929 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:04.750905 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-ldkmf" Sep 19 11:00:04.750999 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:04.750942 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830260-ldkmf" event={"ID":"c85c29cc-2f3c-4ff4-968b-e9d86275eed8","Type":"ContainerDied","Data":"c45987fee24f4cf61019306d9a4199ea3899a1f5df5cd8e6bc2e42f9b1ea8e97"} Sep 19 11:00:04.750999 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:04.750975 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c45987fee24f4cf61019306d9a4199ea3899a1f5df5cd8e6bc2e42f9b1ea8e97" Sep 19 11:00:07.534689 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.534654 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq"] Sep 19 11:00:07.535179 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.534859 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" podUID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerName="istio-proxy" containerID="cri-o://672134461a305a7ca3b65556ef92d0a4d9a761db8e25c5d44c374e503ef659cb" gracePeriod=30 Sep 19 11:00:07.867413 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.867334 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97"] Sep 19 11:00:07.868010 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.867993 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df09fd6b-6881-4c7f-82c4-b39046661850" containerName="cleanup" Sep 19 11:00:07.868073 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.868013 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df09fd6b-6881-4c7f-82c4-b39046661850" containerName="cleanup" Sep 19 11:00:07.868073 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.868036 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53e12cde-d509-47b2-a6c5-f1e25b9db808" containerName="cleanup" Sep 19 11:00:07.868073 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.868041 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="53e12cde-d509-47b2-a6c5-f1e25b9db808" containerName="cleanup" Sep 19 11:00:07.868073 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.868068 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85c29cc-2f3c-4ff4-968b-e9d86275eed8" containerName="cleanup" Sep 19 11:00:07.868073 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.868074 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85c29cc-2f3c-4ff4-968b-e9d86275eed8" containerName="cleanup" Sep 19 11:00:07.868233 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.868081 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf786a48-3f02-4fe9-9915-082985c38b39" containerName="cleanup" Sep 19 11:00:07.868233 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.868086 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf786a48-3f02-4fe9-9915-082985c38b39" containerName="cleanup" Sep 19 11:00:07.868233 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.868146 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df09fd6b-6881-4c7f-82c4-b39046661850" containerName="cleanup" Sep 19 11:00:07.868233 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.868152 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="53e12cde-d509-47b2-a6c5-f1e25b9db808" containerName="cleanup" Sep 19 11:00:07.868233 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.868159 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85c29cc-2f3c-4ff4-968b-e9d86275eed8" containerName="cleanup" Sep 19 11:00:07.868233 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.868165 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf786a48-3f02-4fe9-9915-082985c38b39" containerName="cleanup" Sep 19 11:00:07.871372 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.871354 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97" Sep 19 11:00:07.873586 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.873563 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 11:00:07.873680 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.873571 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 11:00:07.877113 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.877085 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97"] Sep 19 11:00:07.932073 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:07.932035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f367b90-0ac9-41e8-803b-56cad8f3f085-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97\" (UID: \"5f367b90-0ac9-41e8-803b-56cad8f3f085\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97" Sep 19 11:00:08.033446 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:08.033415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f367b90-0ac9-41e8-803b-56cad8f3f085-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97\" (UID: \"5f367b90-0ac9-41e8-803b-56cad8f3f085\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97" Sep 19 11:00:08.033964 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:08.033948 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f367b90-0ac9-41e8-803b-56cad8f3f085-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97\" (UID: \"5f367b90-0ac9-41e8-803b-56cad8f3f085\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97" Sep 19 11:00:08.183739 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:08.183712 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97" Sep 19 11:00:08.470649 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:08.470557 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97"] Sep 19 11:00:08.476821 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:00:08.476783 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f367b90_0ac9_41e8_803b_56cad8f3f085.slice/crio-9059a336695d50bc7bf28e56e5b76ef609e2d1f9f605dfd25d22297d0f439971 WatchSource:0}: Error finding container 9059a336695d50bc7bf28e56e5b76ef609e2d1f9f605dfd25d22297d0f439971: Status 404 returned error can't find the container with id 9059a336695d50bc7bf28e56e5b76ef609e2d1f9f605dfd25d22297d0f439971 Sep 19 11:00:08.776030 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:08.775999 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97" event={"ID":"5f367b90-0ac9-41e8-803b-56cad8f3f085","Type":"ContainerStarted","Data":"5bc396061d2b819e2b00641b70b71c9238087f12d680f57dde562647ce2d4e64"} Sep 19 11:00:08.776406 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:08.776051 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97" event={"ID":"5f367b90-0ac9-41e8-803b-56cad8f3f085","Type":"ContainerStarted","Data":"9059a336695d50bc7bf28e56e5b76ef609e2d1f9f605dfd25d22297d0f439971"} Sep 19 11:00:08.791831 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:08.791781 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97" podStartSLOduration=1.7917652419999999 podStartE2EDuration="1.791765242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:00:08.789698248 +0000 UTC m=+1540.702202970" watchObservedRunningTime="2026-09-19 11:00:08.791765242 +0000 UTC m=+1540.704269958" Sep 19 11:00:09.783672 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:09.783635 2579 generic.go:356] "Generic (PLEG): container finished" podID="5f367b90-0ac9-41e8-803b-56cad8f3f085" containerID="5bc396061d2b819e2b00641b70b71c9238087f12d680f57dde562647ce2d4e64" exitCode=0 Sep 19 11:00:09.784017 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:09.783720 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97" event={"ID":"5f367b90-0ac9-41e8-803b-56cad8f3f085","Type":"ContainerDied","Data":"5bc396061d2b819e2b00641b70b71c9238087f12d680f57dde562647ce2d4e64"} Sep 19 11:00:10.928091 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:10.928070 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97" Sep 19 11:00:11.066834 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:11.066760 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f367b90-0ac9-41e8-803b-56cad8f3f085-maas-api-service-ca\") pod \"5f367b90-0ac9-41e8-803b-56cad8f3f085\" (UID: \"5f367b90-0ac9-41e8-803b-56cad8f3f085\") " Sep 19 11:00:11.067123 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:11.067096 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f367b90-0ac9-41e8-803b-56cad8f3f085-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f367b90-0ac9-41e8-803b-56cad8f3f085" (UID: "5f367b90-0ac9-41e8-803b-56cad8f3f085"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:00:11.168229 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:11.168197 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f367b90-0ac9-41e8-803b-56cad8f3f085-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:11.797476 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:11.797442 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97" Sep 19 11:00:11.797476 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:11.797456 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97" event={"ID":"5f367b90-0ac9-41e8-803b-56cad8f3f085","Type":"ContainerDied","Data":"9059a336695d50bc7bf28e56e5b76ef609e2d1f9f605dfd25d22297d0f439971"} Sep 19 11:00:11.797686 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:11.797498 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9059a336695d50bc7bf28e56e5b76ef609e2d1f9f605dfd25d22297d0f439971" Sep 19 11:00:12.798862 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.798805 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 11:00:12.804622 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.804594 2579 generic.go:356] "Generic (PLEG): container finished" podID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerID="672134461a305a7ca3b65556ef92d0a4d9a761db8e25c5d44c374e503ef659cb" exitCode=0 Sep 19 11:00:12.804751 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.804628 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" event={"ID":"975e4e10-588f-4eeb-af95-ae61197f97e9","Type":"ContainerDied","Data":"672134461a305a7ca3b65556ef92d0a4d9a761db8e25c5d44c374e503ef659cb"} Sep 19 11:00:12.804751 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.804680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" event={"ID":"975e4e10-588f-4eeb-af95-ae61197f97e9","Type":"ContainerDied","Data":"aa593c365b7d42ea3bf478d4e4771312f7f21c940619a3b3b70293753fb6f852"} Sep 19 11:00:12.804751 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.804635 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq" Sep 19 11:00:12.804751 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.804720 2579 scope.go:117] "RemoveContainer" containerID="672134461a305a7ca3b65556ef92d0a4d9a761db8e25c5d44c374e503ef659cb" Sep 19 11:00:12.814900 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.814881 2579 scope.go:117] "RemoveContainer" containerID="31aba3ddf08c1ed34aabcce57d3495a299666d63416336888157b5cd20791d3f" Sep 19 11:00:12.824504 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.824479 2579 scope.go:117] "RemoveContainer" containerID="672134461a305a7ca3b65556ef92d0a4d9a761db8e25c5d44c374e503ef659cb" Sep 19 11:00:12.824830 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:00:12.824810 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"672134461a305a7ca3b65556ef92d0a4d9a761db8e25c5d44c374e503ef659cb\": container with ID starting with 672134461a305a7ca3b65556ef92d0a4d9a761db8e25c5d44c374e503ef659cb not found: ID does not exist" containerID="672134461a305a7ca3b65556ef92d0a4d9a761db8e25c5d44c374e503ef659cb" Sep 19 11:00:12.824918 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.824845 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672134461a305a7ca3b65556ef92d0a4d9a761db8e25c5d44c374e503ef659cb"} err="failed to get container status \"672134461a305a7ca3b65556ef92d0a4d9a761db8e25c5d44c374e503ef659cb\": rpc error: code = NotFound desc = could not find container \"672134461a305a7ca3b65556ef92d0a4d9a761db8e25c5d44c374e503ef659cb\": container with ID starting with 672134461a305a7ca3b65556ef92d0a4d9a761db8e25c5d44c374e503ef659cb not found: ID does not exist" Sep 19 11:00:12.824918 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.824862 2579 scope.go:117] "RemoveContainer" containerID="31aba3ddf08c1ed34aabcce57d3495a299666d63416336888157b5cd20791d3f" Sep 19 11:00:12.825131 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:00:12.825113 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31aba3ddf08c1ed34aabcce57d3495a299666d63416336888157b5cd20791d3f\": container with ID starting with 31aba3ddf08c1ed34aabcce57d3495a299666d63416336888157b5cd20791d3f not found: ID does not exist" containerID="31aba3ddf08c1ed34aabcce57d3495a299666d63416336888157b5cd20791d3f" Sep 19 11:00:12.825210 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.825135 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31aba3ddf08c1ed34aabcce57d3495a299666d63416336888157b5cd20791d3f"} err="failed to get container status \"31aba3ddf08c1ed34aabcce57d3495a299666d63416336888157b5cd20791d3f\": rpc error: code = NotFound desc = could not find container \"31aba3ddf08c1ed34aabcce57d3495a299666d63416336888157b5cd20791d3f\": container with ID starting with 31aba3ddf08c1ed34aabcce57d3495a299666d63416336888157b5cd20791d3f not found: ID does not exist" Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.883331 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-workload-socket\") pod \"975e4e10-588f-4eeb-af95-ae61197f97e9\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.883406 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975e4e10-588f-4eeb-af95-ae61197f97e9-istiod-ca-cert\") pod \"975e4e10-588f-4eeb-af95-ae61197f97e9\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.883495 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-token\") pod \"975e4e10-588f-4eeb-af95-ae61197f97e9\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.883571 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-data\") pod \"975e4e10-588f-4eeb-af95-ae61197f97e9\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.883605 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kjp2\" (UniqueName: \"kubernetes.io/projected/975e4e10-588f-4eeb-af95-ae61197f97e9-kube-api-access-8kjp2\") pod \"975e4e10-588f-4eeb-af95-ae61197f97e9\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.883690 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-podinfo\") pod \"975e4e10-588f-4eeb-af95-ae61197f97e9\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.883721 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-envoy\") pod \"975e4e10-588f-4eeb-af95-ae61197f97e9\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.883751 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "975e4e10-588f-4eeb-af95-ae61197f97e9" (UID: "975e4e10-588f-4eeb-af95-ae61197f97e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.883778 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-workload-certs\") pod \"975e4e10-588f-4eeb-af95-ae61197f97e9\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.883826 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-credential-socket\") pod \"975e4e10-588f-4eeb-af95-ae61197f97e9\" (UID: \"975e4e10-588f-4eeb-af95-ae61197f97e9\") " Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.883934 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-data" (OuterVolumeSpecName: "istio-data") pod "975e4e10-588f-4eeb-af95-ae61197f97e9" (UID: "975e4e10-588f-4eeb-af95-ae61197f97e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.884266 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975e4e10-588f-4eeb-af95-ae61197f97e9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "975e4e10-588f-4eeb-af95-ae61197f97e9" (UID: "975e4e10-588f-4eeb-af95-ae61197f97e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.884382 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.884406 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975e4e10-588f-4eeb-af95-ae61197f97e9-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:12.884466 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.884420 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:12.885616 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.885321 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "975e4e10-588f-4eeb-af95-ae61197f97e9" (UID: "975e4e10-588f-4eeb-af95-ae61197f97e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:12.886973 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.886946 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-token" (OuterVolumeSpecName: "istio-token") pod "975e4e10-588f-4eeb-af95-ae61197f97e9" (UID: "975e4e10-588f-4eeb-af95-ae61197f97e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:00:12.889104 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.889047 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "975e4e10-588f-4eeb-af95-ae61197f97e9" (UID: "975e4e10-588f-4eeb-af95-ae61197f97e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:12.891268 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.891241 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "975e4e10-588f-4eeb-af95-ae61197f97e9" (UID: "975e4e10-588f-4eeb-af95-ae61197f97e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 11:00:12.891460 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.891440 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "975e4e10-588f-4eeb-af95-ae61197f97e9" (UID: "975e4e10-588f-4eeb-af95-ae61197f97e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:12.891830 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.891804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975e4e10-588f-4eeb-af95-ae61197f97e9-kube-api-access-8kjp2" (OuterVolumeSpecName: "kube-api-access-8kjp2") pod "975e4e10-588f-4eeb-af95-ae61197f97e9" (UID: "975e4e10-588f-4eeb-af95-ae61197f97e9"). InnerVolumeSpecName "kube-api-access-8kjp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:00:12.985813 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.985781 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:12.985813 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.985809 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kjp2\" (UniqueName: \"kubernetes.io/projected/975e4e10-588f-4eeb-af95-ae61197f97e9-kube-api-access-8kjp2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:12.985975 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.985822 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:12.985975 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.985830 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:12.985975 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.985838 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:12.985975 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:12.985846 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975e4e10-588f-4eeb-af95-ae61197f97e9-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:13.127121 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:13.127091 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq"] Sep 19 11:00:13.131339 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:13.131312 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq"] Sep 19 11:00:14.196374 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:14.196341 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="975e4e10-588f-4eeb-af95-ae61197f97e9" path="/var/lib/kubelet/pods/975e4e10-588f-4eeb-af95-ae61197f97e9/volumes" Sep 19 11:00:18.599951 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:18.599915 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97"] Sep 19 11:00:18.604765 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:18.604736 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-9ac6b3-czk97"] Sep 19 11:00:20.196667 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:20.196635 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f367b90-0ac9-41e8-803b-56cad8f3f085" path="/var/lib/kubelet/pods/5f367b90-0ac9-41e8-803b-56cad8f3f085/volumes" Sep 19 11:00:25.567092 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.567037 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d"] Sep 19 11:00:25.567565 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.567358 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" podUID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerName="istio-proxy" containerID="cri-o://e3cb1288f5e9279764d059a9479f56c808bf39e8560a4803782833814e823ea7" gracePeriod=30 Sep 19 11:00:25.917276 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.917241 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2"] Sep 19 11:00:25.917670 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.917658 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerName="istio-proxy" Sep 19 11:00:25.917722 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.917671 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerName="istio-proxy" Sep 19 11:00:25.917722 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.917680 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f367b90-0ac9-41e8-803b-56cad8f3f085" containerName="revoke-keys" Sep 19 11:00:25.917722 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.917686 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f367b90-0ac9-41e8-803b-56cad8f3f085" containerName="revoke-keys" Sep 19 11:00:25.917722 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.917700 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerName="istio-proxy" Sep 19 11:00:25.917722 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.917706 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerName="istio-proxy" Sep 19 11:00:25.917879 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.917766 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerName="istio-proxy" Sep 19 11:00:25.917879 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.917775 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f367b90-0ac9-41e8-803b-56cad8f3f085" containerName="revoke-keys" Sep 19 11:00:25.921029 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.921013 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2" Sep 19 11:00:25.923266 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.923248 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 11:00:25.923451 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.923439 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 11:00:25.927543 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:25.927521 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2"] Sep 19 11:00:26.018376 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:26.018348 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d43c84c-a809-4750-8346-16301e92019c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2\" (UID: \"7d43c84c-a809-4750-8346-16301e92019c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2" Sep 19 11:00:26.119476 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:26.119439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d43c84c-a809-4750-8346-16301e92019c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2\" (UID: \"7d43c84c-a809-4750-8346-16301e92019c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2" Sep 19 11:00:26.120014 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:26.119997 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d43c84c-a809-4750-8346-16301e92019c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2\" (UID: \"7d43c84c-a809-4750-8346-16301e92019c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2" Sep 19 11:00:26.235201 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:26.235123 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2" Sep 19 11:00:26.505937 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:26.505904 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2"] Sep 19 11:00:26.508137 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:00:26.508107 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d43c84c_a809_4750_8346_16301e92019c.slice/crio-785e85b1a4ab75a4f1e13209088ad7f9c9a3629e8f55aeb8fcd0e67e1a68a516 WatchSource:0}: Error finding container 785e85b1a4ab75a4f1e13209088ad7f9c9a3629e8f55aeb8fcd0e67e1a68a516: Status 404 returned error can't find the container with id 785e85b1a4ab75a4f1e13209088ad7f9c9a3629e8f55aeb8fcd0e67e1a68a516 Sep 19 11:00:26.887463 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:26.887411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2" event={"ID":"7d43c84c-a809-4750-8346-16301e92019c","Type":"ContainerStarted","Data":"47a353c0c76a9ff43290618dc627f392c710ff1e92d3f3e6798dc04812bfcfcf"} Sep 19 11:00:26.887888 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:26.887480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2" event={"ID":"7d43c84c-a809-4750-8346-16301e92019c","Type":"ContainerStarted","Data":"785e85b1a4ab75a4f1e13209088ad7f9c9a3629e8f55aeb8fcd0e67e1a68a516"} Sep 19 11:00:26.902242 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:26.902169 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2" podStartSLOduration=1.902152885 podStartE2EDuration="1.902152885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:00:26.901149962 +0000 UTC m=+1558.813654680" watchObservedRunningTime="2026-09-19 11:00:26.902152885 +0000 UTC m=+1558.814657600" Sep 19 11:00:27.894333 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:27.894302 2579 generic.go:356] "Generic (PLEG): container finished" podID="7d43c84c-a809-4750-8346-16301e92019c" containerID="47a353c0c76a9ff43290618dc627f392c710ff1e92d3f3e6798dc04812bfcfcf" exitCode=0 Sep 19 11:00:27.894721 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:27.894339 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2" event={"ID":"7d43c84c-a809-4750-8346-16301e92019c","Type":"ContainerDied","Data":"47a353c0c76a9ff43290618dc627f392c710ff1e92d3f3e6798dc04812bfcfcf"} Sep 19 11:00:29.023102 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:29.023069 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2" Sep 19 11:00:29.147589 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:29.147544 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d43c84c-a809-4750-8346-16301e92019c-maas-api-service-ca\") pod \"7d43c84c-a809-4750-8346-16301e92019c\" (UID: \"7d43c84c-a809-4750-8346-16301e92019c\") " Sep 19 11:00:29.147941 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:29.147921 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d43c84c-a809-4750-8346-16301e92019c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7d43c84c-a809-4750-8346-16301e92019c" (UID: "7d43c84c-a809-4750-8346-16301e92019c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:00:29.248681 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:29.248580 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d43c84c-a809-4750-8346-16301e92019c-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:29.906808 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:29.906777 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2" Sep 19 11:00:29.906808 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:29.906789 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2" event={"ID":"7d43c84c-a809-4750-8346-16301e92019c","Type":"ContainerDied","Data":"785e85b1a4ab75a4f1e13209088ad7f9c9a3629e8f55aeb8fcd0e67e1a68a516"} Sep 19 11:00:29.907011 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:29.906835 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="785e85b1a4ab75a4f1e13209088ad7f9c9a3629e8f55aeb8fcd0e67e1a68a516" Sep 19 11:00:30.826303 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.826275 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 11:00:30.858745 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.858716 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-workload-socket\") pod \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " Sep 19 11:00:30.858876 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.858780 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-credential-socket\") pod \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " Sep 19 11:00:30.858876 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.858818 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-workload-certs\") pod \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " Sep 19 11:00:30.858876 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.858849 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-data\") pod \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " Sep 19 11:00:30.859010 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.858897 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-podinfo\") pod \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " Sep 19 11:00:30.859010 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.858928 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-envoy\") pod \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " Sep 19 11:00:30.859010 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.858974 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istiod-ca-cert\") pod \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " Sep 19 11:00:30.859171 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.859008 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkvgk\" (UniqueName: \"kubernetes.io/projected/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-kube-api-access-jkvgk\") pod \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " Sep 19 11:00:30.859171 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.859072 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" (UID: "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:30.859171 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.859049 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-token\") pod \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\" (UID: \"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8\") " Sep 19 11:00:30.859327 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.859165 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" (UID: "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:30.859327 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.859260 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-data" (OuterVolumeSpecName: "istio-data") pod "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" (UID: "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:30.859439 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.859350 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" (UID: "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:00:30.859439 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.859034 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" (UID: "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:30.859559 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.859523 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:30.859559 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.859539 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:30.859559 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.859549 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:30.859664 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.859565 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:30.859664 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.859585 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:30.861389 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.861358 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" (UID: "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:30.861497 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.861406 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-token" (OuterVolumeSpecName: "istio-token") pod "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" (UID: "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:00:30.861625 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.861607 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-kube-api-access-jkvgk" (OuterVolumeSpecName: "kube-api-access-jkvgk") pod "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" (UID: "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8"). InnerVolumeSpecName "kube-api-access-jkvgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:00:30.861718 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.861702 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" (UID: "351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 11:00:30.915116 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.915088 2579 generic.go:356] "Generic (PLEG): container finished" podID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerID="e3cb1288f5e9279764d059a9479f56c808bf39e8560a4803782833814e823ea7" exitCode=0 Sep 19 11:00:30.915250 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.915189 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" Sep 19 11:00:30.915289 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.915190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" event={"ID":"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8","Type":"ContainerDied","Data":"e3cb1288f5e9279764d059a9479f56c808bf39e8560a4803782833814e823ea7"} Sep 19 11:00:30.915333 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.915313 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d" event={"ID":"351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8","Type":"ContainerDied","Data":"b9244664519dff300d2f5fde40aca72d1bbea9f7508b327ef4cf800e460ac02a"} Sep 19 11:00:30.915374 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.915350 2579 scope.go:117] "RemoveContainer" containerID="e3cb1288f5e9279764d059a9479f56c808bf39e8560a4803782833814e823ea7" Sep 19 11:00:30.924604 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.924589 2579 scope.go:117] "RemoveContainer" containerID="f5a4cb03009f987c99a4691b0b02623b513d3e492049a8667bb359d39f6e3018" Sep 19 11:00:30.932650 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.932635 2579 scope.go:117] "RemoveContainer" containerID="e3cb1288f5e9279764d059a9479f56c808bf39e8560a4803782833814e823ea7" Sep 19 11:00:30.933102 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:00:30.933073 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3cb1288f5e9279764d059a9479f56c808bf39e8560a4803782833814e823ea7\": container with ID starting with e3cb1288f5e9279764d059a9479f56c808bf39e8560a4803782833814e823ea7 not found: ID does not exist" containerID="e3cb1288f5e9279764d059a9479f56c808bf39e8560a4803782833814e823ea7" Sep 19 11:00:30.933290 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.933112 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3cb1288f5e9279764d059a9479f56c808bf39e8560a4803782833814e823ea7"} err="failed to get container status \"e3cb1288f5e9279764d059a9479f56c808bf39e8560a4803782833814e823ea7\": rpc error: code = NotFound desc = could not find container \"e3cb1288f5e9279764d059a9479f56c808bf39e8560a4803782833814e823ea7\": container with ID starting with e3cb1288f5e9279764d059a9479f56c808bf39e8560a4803782833814e823ea7 not found: ID does not exist" Sep 19 11:00:30.933290 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.933157 2579 scope.go:117] "RemoveContainer" containerID="f5a4cb03009f987c99a4691b0b02623b513d3e492049a8667bb359d39f6e3018" Sep 19 11:00:30.934164 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:00:30.934144 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a4cb03009f987c99a4691b0b02623b513d3e492049a8667bb359d39f6e3018\": container with ID starting with f5a4cb03009f987c99a4691b0b02623b513d3e492049a8667bb359d39f6e3018 not found: ID does not exist" containerID="f5a4cb03009f987c99a4691b0b02623b513d3e492049a8667bb359d39f6e3018" Sep 19 11:00:30.934240 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.934169 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a4cb03009f987c99a4691b0b02623b513d3e492049a8667bb359d39f6e3018"} err="failed to get container status \"f5a4cb03009f987c99a4691b0b02623b513d3e492049a8667bb359d39f6e3018\": rpc error: code = NotFound desc = could not find container \"f5a4cb03009f987c99a4691b0b02623b513d3e492049a8667bb359d39f6e3018\": container with ID starting with f5a4cb03009f987c99a4691b0b02623b513d3e492049a8667bb359d39f6e3018 not found: ID does not exist" Sep 19 11:00:30.936362 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.936341 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d"] Sep 19 11:00:30.940146 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.940122 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d"] Sep 19 11:00:30.960033 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.960012 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:30.960033 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.960035 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:30.960169 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.960044 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkvgk\" (UniqueName: \"kubernetes.io/projected/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-kube-api-access-jkvgk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:30.960169 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:30.960080 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:32.196476 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:32.196445 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" path="/var/lib/kubelet/pods/351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8/volumes" Sep 19 11:00:36.337733 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.337699 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8"] Sep 19 11:00:36.340951 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.340922 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-266c89-29830260-99pc8"] Sep 19 11:00:36.412629 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.412596 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl"] Sep 19 11:00:36.413094 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.413034 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" podUID="c24815e6-1b76-4f80-80bb-f3b5c25c866a" containerName="payload-pre-processing" containerID="cri-o://2bc13e303f0dc54aa8e47b48948f42761fa275482884649f1b791a8b16382419" gracePeriod=30 Sep 19 11:00:36.594473 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.594395 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2"] Sep 19 11:00:36.596803 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.596778 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-266c89-6ddz2"] Sep 19 11:00:36.668502 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.668471 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" Sep 19 11:00:36.712936 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.712907 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24815e6-1b76-4f80-80bb-f3b5c25c866a-plugins-config-volume\") pod \"c24815e6-1b76-4f80-80bb-f3b5c25c866a\" (UID: \"c24815e6-1b76-4f80-80bb-f3b5c25c866a\") " Sep 19 11:00:36.713113 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.713006 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcrvn\" (UniqueName: \"kubernetes.io/projected/c24815e6-1b76-4f80-80bb-f3b5c25c866a-kube-api-access-dcrvn\") pod \"c24815e6-1b76-4f80-80bb-f3b5c25c866a\" (UID: \"c24815e6-1b76-4f80-80bb-f3b5c25c866a\") " Sep 19 11:00:36.713364 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.713339 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24815e6-1b76-4f80-80bb-f3b5c25c866a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c24815e6-1b76-4f80-80bb-f3b5c25c866a" (UID: "c24815e6-1b76-4f80-80bb-f3b5c25c866a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:00:36.714997 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.714979 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24815e6-1b76-4f80-80bb-f3b5c25c866a-kube-api-access-dcrvn" (OuterVolumeSpecName: "kube-api-access-dcrvn") pod "c24815e6-1b76-4f80-80bb-f3b5c25c866a" (UID: "c24815e6-1b76-4f80-80bb-f3b5c25c866a"). InnerVolumeSpecName "kube-api-access-dcrvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:00:36.813755 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.813727 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24815e6-1b76-4f80-80bb-f3b5c25c866a-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:36.813755 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.813752 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcrvn\" (UniqueName: \"kubernetes.io/projected/c24815e6-1b76-4f80-80bb-f3b5c25c866a-kube-api-access-dcrvn\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:36.960784 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.960752 2579 generic.go:356] "Generic (PLEG): container finished" podID="c24815e6-1b76-4f80-80bb-f3b5c25c866a" containerID="2bc13e303f0dc54aa8e47b48948f42761fa275482884649f1b791a8b16382419" exitCode=0 Sep 19 11:00:36.960947 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.960798 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" Sep 19 11:00:36.960947 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.960836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" event={"ID":"c24815e6-1b76-4f80-80bb-f3b5c25c866a","Type":"ContainerDied","Data":"2bc13e303f0dc54aa8e47b48948f42761fa275482884649f1b791a8b16382419"} Sep 19 11:00:36.960947 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.960890 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl" event={"ID":"c24815e6-1b76-4f80-80bb-f3b5c25c866a","Type":"ContainerDied","Data":"5c27649c6c4fa9210776a0ef97938f1e30dd61ff4833ada1e11a4a2ec8ad5de3"} Sep 19 11:00:36.960947 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.960915 2579 scope.go:117] "RemoveContainer" containerID="2bc13e303f0dc54aa8e47b48948f42761fa275482884649f1b791a8b16382419" Sep 19 11:00:36.971333 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.971317 2579 scope.go:117] "RemoveContainer" containerID="2bc13e303f0dc54aa8e47b48948f42761fa275482884649f1b791a8b16382419" Sep 19 11:00:36.971602 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:00:36.971584 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc13e303f0dc54aa8e47b48948f42761fa275482884649f1b791a8b16382419\": container with ID starting with 2bc13e303f0dc54aa8e47b48948f42761fa275482884649f1b791a8b16382419 not found: ID does not exist" containerID="2bc13e303f0dc54aa8e47b48948f42761fa275482884649f1b791a8b16382419" Sep 19 11:00:36.971654 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.971610 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc13e303f0dc54aa8e47b48948f42761fa275482884649f1b791a8b16382419"} err="failed to get container status \"2bc13e303f0dc54aa8e47b48948f42761fa275482884649f1b791a8b16382419\": rpc error: code = NotFound desc = could not find container \"2bc13e303f0dc54aa8e47b48948f42761fa275482884649f1b791a8b16382419\": container with ID starting with 2bc13e303f0dc54aa8e47b48948f42761fa275482884649f1b791a8b16382419 not found: ID does not exist" Sep 19 11:00:36.981734 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.981712 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl"] Sep 19 11:00:36.985264 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:36.985242 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl"] Sep 19 11:00:38.195855 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:38.195812 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d43c84c-a809-4750-8346-16301e92019c" path="/var/lib/kubelet/pods/7d43c84c-a809-4750-8346-16301e92019c/volumes" Sep 19 11:00:38.196393 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:38.196373 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c24815e6-1b76-4f80-80bb-f3b5c25c866a" path="/var/lib/kubelet/pods/c24815e6-1b76-4f80-80bb-f3b5c25c866a/volumes" Sep 19 11:00:38.196702 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:38.196684 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df09fd6b-6881-4c7f-82c4-b39046661850" path="/var/lib/kubelet/pods/df09fd6b-6881-4c7f-82c4-b39046661850/volumes" Sep 19 11:00:42.707422 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:42.707387 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j"] Sep 19 11:00:42.707997 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:42.707637 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" podUID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerName="istio-proxy" containerID="cri-o://4c9790ba95de72d079a9eeb9414dd24b79fb570ea6674d317208976d73a96486" gracePeriod=30 Sep 19 11:00:47.959589 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:47.959567 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 11:00:48.019546 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.019481 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istiod-ca-cert\") pod \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " Sep 19 11:00:48.019668 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.019546 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-workload-socket\") pod \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " Sep 19 11:00:48.019729 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.019707 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-credential-socket\") pod \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " Sep 19 11:00:48.019767 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.019748 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-podinfo\") pod \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " Sep 19 11:00:48.019811 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.019779 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-workload-certs\") pod \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " Sep 19 11:00:48.019811 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.019787 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c33bf4e-8864-4014-90e5-cb5cf43a07b6" (UID: "0c33bf4e-8864-4014-90e5-cb5cf43a07b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:48.019900 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.019815 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-data\") pod \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " Sep 19 11:00:48.019900 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.019856 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-envoy\") pod \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " Sep 19 11:00:48.019900 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.019869 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c33bf4e-8864-4014-90e5-cb5cf43a07b6" (UID: "0c33bf4e-8864-4014-90e5-cb5cf43a07b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:00:48.019900 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.019884 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-token\") pod \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " Sep 19 11:00:48.020134 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.019947 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5gw4\" (UniqueName: \"kubernetes.io/projected/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-kube-api-access-t5gw4\") pod \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\" (UID: \"0c33bf4e-8864-4014-90e5-cb5cf43a07b6\") " Sep 19 11:00:48.020134 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.019988 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c33bf4e-8864-4014-90e5-cb5cf43a07b6" (UID: "0c33bf4e-8864-4014-90e5-cb5cf43a07b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:48.020134 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.020105 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c33bf4e-8864-4014-90e5-cb5cf43a07b6" (UID: "0c33bf4e-8864-4014-90e5-cb5cf43a07b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:48.020302 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.020141 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c33bf4e-8864-4014-90e5-cb5cf43a07b6" (UID: "0c33bf4e-8864-4014-90e5-cb5cf43a07b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:48.020363 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.020341 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:48.020424 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.020362 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:48.020424 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.020376 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:48.020424 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.020388 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:48.020424 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.020408 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:48.022530 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.022385 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c33bf4e-8864-4014-90e5-cb5cf43a07b6" (UID: "0c33bf4e-8864-4014-90e5-cb5cf43a07b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:00:48.022946 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.022531 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c33bf4e-8864-4014-90e5-cb5cf43a07b6" (UID: "0c33bf4e-8864-4014-90e5-cb5cf43a07b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 11:00:48.022946 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.022654 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-kube-api-access-t5gw4" (OuterVolumeSpecName: "kube-api-access-t5gw4") pod "0c33bf4e-8864-4014-90e5-cb5cf43a07b6" (UID: "0c33bf4e-8864-4014-90e5-cb5cf43a07b6"). InnerVolumeSpecName "kube-api-access-t5gw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:00:48.022946 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.022668 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c33bf4e-8864-4014-90e5-cb5cf43a07b6" (UID: "0c33bf4e-8864-4014-90e5-cb5cf43a07b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:00:48.029108 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.029084 2579 generic.go:356] "Generic (PLEG): container finished" podID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerID="4c9790ba95de72d079a9eeb9414dd24b79fb570ea6674d317208976d73a96486" exitCode=0 Sep 19 11:00:48.029222 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.029116 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" Sep 19 11:00:48.029222 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.029144 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" event={"ID":"0c33bf4e-8864-4014-90e5-cb5cf43a07b6","Type":"ContainerDied","Data":"4c9790ba95de72d079a9eeb9414dd24b79fb570ea6674d317208976d73a96486"} Sep 19 11:00:48.029222 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.029184 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j" event={"ID":"0c33bf4e-8864-4014-90e5-cb5cf43a07b6","Type":"ContainerDied","Data":"7a583e101b3ccdead066421ca052d5fe469547bdd71c33e94fa537971ae1bc7e"} Sep 19 11:00:48.029222 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.029206 2579 scope.go:117] "RemoveContainer" containerID="4c9790ba95de72d079a9eeb9414dd24b79fb570ea6674d317208976d73a96486" Sep 19 11:00:48.045192 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.045171 2579 scope.go:117] "RemoveContainer" containerID="d87a47f6a811a0dc5f883be378a6c8ecb66974ae8ec23a5a1d4b93260b37309c" Sep 19 11:00:48.056093 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.056074 2579 scope.go:117] "RemoveContainer" containerID="4c9790ba95de72d079a9eeb9414dd24b79fb570ea6674d317208976d73a96486" Sep 19 11:00:48.056304 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.056174 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j"] Sep 19 11:00:48.056557 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:00:48.056537 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9790ba95de72d079a9eeb9414dd24b79fb570ea6674d317208976d73a96486\": container with ID starting with 4c9790ba95de72d079a9eeb9414dd24b79fb570ea6674d317208976d73a96486 not found: ID does not exist" containerID="4c9790ba95de72d079a9eeb9414dd24b79fb570ea6674d317208976d73a96486" Sep 19 11:00:48.056634 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.056565 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9790ba95de72d079a9eeb9414dd24b79fb570ea6674d317208976d73a96486"} err="failed to get container status \"4c9790ba95de72d079a9eeb9414dd24b79fb570ea6674d317208976d73a96486\": rpc error: code = NotFound desc = could not find container \"4c9790ba95de72d079a9eeb9414dd24b79fb570ea6674d317208976d73a96486\": container with ID starting with 4c9790ba95de72d079a9eeb9414dd24b79fb570ea6674d317208976d73a96486 not found: ID does not exist" Sep 19 11:00:48.056634 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.056580 2579 scope.go:117] "RemoveContainer" containerID="d87a47f6a811a0dc5f883be378a6c8ecb66974ae8ec23a5a1d4b93260b37309c" Sep 19 11:00:48.056867 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:00:48.056848 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87a47f6a811a0dc5f883be378a6c8ecb66974ae8ec23a5a1d4b93260b37309c\": container with ID starting with d87a47f6a811a0dc5f883be378a6c8ecb66974ae8ec23a5a1d4b93260b37309c not found: ID does not exist" containerID="d87a47f6a811a0dc5f883be378a6c8ecb66974ae8ec23a5a1d4b93260b37309c" Sep 19 11:00:48.056930 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.056870 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87a47f6a811a0dc5f883be378a6c8ecb66974ae8ec23a5a1d4b93260b37309c"} err="failed to get container status \"d87a47f6a811a0dc5f883be378a6c8ecb66974ae8ec23a5a1d4b93260b37309c\": rpc error: code = NotFound desc = could not find container \"d87a47f6a811a0dc5f883be378a6c8ecb66974ae8ec23a5a1d4b93260b37309c\": container with ID starting with d87a47f6a811a0dc5f883be378a6c8ecb66974ae8ec23a5a1d4b93260b37309c not found: ID does not exist" Sep 19 11:00:48.058470 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.058447 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j"] Sep 19 11:00:48.121103 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.121082 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:48.121103 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.121101 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:48.121246 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.121112 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:48.121246 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.121120 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5gw4\" (UniqueName: \"kubernetes.io/projected/0c33bf4e-8864-4014-90e5-cb5cf43a07b6-kube-api-access-t5gw4\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:00:48.196229 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:00:48.196203 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" path="/var/lib/kubelet/pods/0c33bf4e-8864-4014-90e5-cb5cf43a07b6/volumes" Sep 19 11:01:30.521829 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:30.521796 2579 scope.go:117] "RemoveContainer" containerID="b2dbe0f8039fd3bf09e0474934a6d0048bde26b89cdfa37317f692480b09cfec" Sep 19 11:01:43.324669 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.324635 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn"] Sep 19 11:01:43.325028 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325022 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d43c84c-a809-4750-8346-16301e92019c" containerName="revoke-keys" Sep 19 11:01:43.325101 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325032 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d43c84c-a809-4750-8346-16301e92019c" containerName="revoke-keys" Sep 19 11:01:43.325101 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325046 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerName="istio-proxy" Sep 19 11:01:43.325101 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325052 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerName="istio-proxy" Sep 19 11:01:43.325101 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325082 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerName="istio-proxy" Sep 19 11:01:43.325101 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325090 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerName="istio-proxy" Sep 19 11:01:43.325101 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325098 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerName="istio-proxy" Sep 19 11:01:43.325101 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325103 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerName="istio-proxy" Sep 19 11:01:43.325340 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325125 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24815e6-1b76-4f80-80bb-f3b5c25c866a" containerName="payload-pre-processing" Sep 19 11:01:43.325340 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325131 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24815e6-1b76-4f80-80bb-f3b5c25c866a" containerName="payload-pre-processing" Sep 19 11:01:43.325340 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325189 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerName="istio-proxy" Sep 19 11:01:43.325340 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325196 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c24815e6-1b76-4f80-80bb-f3b5c25c866a" containerName="payload-pre-processing" Sep 19 11:01:43.325340 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325204 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerName="istio-proxy" Sep 19 11:01:43.325340 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325209 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c33bf4e-8864-4014-90e5-cb5cf43a07b6" containerName="istio-proxy" Sep 19 11:01:43.325340 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325216 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="975e4e10-588f-4eeb-af95-ae61197f97e9" containerName="istio-proxy" Sep 19 11:01:43.325340 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.325221 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d43c84c-a809-4750-8346-16301e92019c" containerName="revoke-keys" Sep 19 11:01:43.329672 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.329656 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" Sep 19 11:01:43.332429 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.332403 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f210d9-dockercfg-6ghgc\"" Sep 19 11:01:43.332567 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.332446 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f210d9\"" Sep 19 11:01:43.335888 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.335867 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn"] Sep 19 11:01:43.366896 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.366868 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j"] Sep 19 11:01:43.367332 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.367315 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerName="istio-proxy" Sep 19 11:01:43.367405 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.367336 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerName="istio-proxy" Sep 19 11:01:43.367444 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.367422 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="351bfa27-6ecf-4ad7-98d9-2d2a1a7027a8" containerName="istio-proxy" Sep 19 11:01:43.370608 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.370588 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" Sep 19 11:01:43.380014 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.379994 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j"] Sep 19 11:01:43.432029 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.431991 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jfhq\" (UniqueName: \"kubernetes.io/projected/01b78706-202f-439a-826a-ba9b2db6a55c-kube-api-access-6jfhq\") pod \"payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn\" (UID: \"01b78706-202f-439a-826a-ba9b2db6a55c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" Sep 19 11:01:43.432236 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.432175 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b78706-202f-439a-826a-ba9b2db6a55c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn\" (UID: \"01b78706-202f-439a-826a-ba9b2db6a55c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" Sep 19 11:01:43.533077 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.533021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe20919-b1cb-4568-bd6c-381f584f55db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j\" (UID: \"8fe20919-b1cb-4568-bd6c-381f584f55db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" Sep 19 11:01:43.533242 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.533087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7lc\" (UniqueName: \"kubernetes.io/projected/8fe20919-b1cb-4568-bd6c-381f584f55db-kube-api-access-wk7lc\") pod \"payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j\" (UID: \"8fe20919-b1cb-4568-bd6c-381f584f55db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" Sep 19 11:01:43.533242 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.533151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b78706-202f-439a-826a-ba9b2db6a55c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn\" (UID: \"01b78706-202f-439a-826a-ba9b2db6a55c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" Sep 19 11:01:43.533242 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.533235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfhq\" (UniqueName: \"kubernetes.io/projected/01b78706-202f-439a-826a-ba9b2db6a55c-kube-api-access-6jfhq\") pod \"payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn\" (UID: \"01b78706-202f-439a-826a-ba9b2db6a55c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" Sep 19 11:01:43.533912 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.533890 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b78706-202f-439a-826a-ba9b2db6a55c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn\" (UID: \"01b78706-202f-439a-826a-ba9b2db6a55c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" Sep 19 11:01:43.541803 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.541777 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jfhq\" (UniqueName: \"kubernetes.io/projected/01b78706-202f-439a-826a-ba9b2db6a55c-kube-api-access-6jfhq\") pod \"payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn\" (UID: \"01b78706-202f-439a-826a-ba9b2db6a55c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" Sep 19 11:01:43.633951 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.633928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe20919-b1cb-4568-bd6c-381f584f55db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j\" (UID: \"8fe20919-b1cb-4568-bd6c-381f584f55db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" Sep 19 11:01:43.634151 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.633966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7lc\" (UniqueName: \"kubernetes.io/projected/8fe20919-b1cb-4568-bd6c-381f584f55db-kube-api-access-wk7lc\") pod \"payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j\" (UID: \"8fe20919-b1cb-4568-bd6c-381f584f55db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" Sep 19 11:01:43.634681 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.634657 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe20919-b1cb-4568-bd6c-381f584f55db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j\" (UID: \"8fe20919-b1cb-4568-bd6c-381f584f55db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" Sep 19 11:01:43.641699 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.641675 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7lc\" (UniqueName: \"kubernetes.io/projected/8fe20919-b1cb-4568-bd6c-381f584f55db-kube-api-access-wk7lc\") pod \"payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j\" (UID: \"8fe20919-b1cb-4568-bd6c-381f584f55db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" Sep 19 11:01:43.642429 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.642402 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" Sep 19 11:01:43.681566 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.681534 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" Sep 19 11:01:43.916235 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.916212 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn"] Sep 19 11:01:43.917775 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:01:43.917745 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b78706_202f_439a_826a_ba9b2db6a55c.slice/crio-48781bee1e23e140b87755a6622fc4c9f5c610f57f49954b30773e8f3e799250 WatchSource:0}: Error finding container 48781bee1e23e140b87755a6622fc4c9f5c610f57f49954b30773e8f3e799250: Status 404 returned error can't find the container with id 48781bee1e23e140b87755a6622fc4c9f5c610f57f49954b30773e8f3e799250 Sep 19 11:01:43.962702 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:43.962620 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j"] Sep 19 11:01:43.967819 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:01:43.967789 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe20919_b1cb_4568_bd6c_381f584f55db.slice/crio-59057ca6aa4971819a906bd481f3e20a6d4e455ab2ec5d5375651ff3e70ce00e WatchSource:0}: Error finding container 59057ca6aa4971819a906bd481f3e20a6d4e455ab2ec5d5375651ff3e70ce00e: Status 404 returned error can't find the container with id 59057ca6aa4971819a906bd481f3e20a6d4e455ab2ec5d5375651ff3e70ce00e Sep 19 11:01:44.357198 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:44.357093 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" event={"ID":"01b78706-202f-439a-826a-ba9b2db6a55c","Type":"ContainerStarted","Data":"91dca0bd89dae46978708cb449081e206601ef0703f05b2cd95a2bebbfacddba"} Sep 19 11:01:44.357198 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:44.357163 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" event={"ID":"01b78706-202f-439a-826a-ba9b2db6a55c","Type":"ContainerStarted","Data":"48781bee1e23e140b87755a6622fc4c9f5c610f57f49954b30773e8f3e799250"} Sep 19 11:01:44.357729 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:44.357209 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" Sep 19 11:01:44.359140 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:44.359111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" event={"ID":"8fe20919-b1cb-4568-bd6c-381f584f55db","Type":"ContainerStarted","Data":"ca2d92413e960591baea15215e75daec154f961e9c8613a01c0aa3d075f232fa"} Sep 19 11:01:44.359293 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:44.359163 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" event={"ID":"8fe20919-b1cb-4568-bd6c-381f584f55db","Type":"ContainerStarted","Data":"59057ca6aa4971819a906bd481f3e20a6d4e455ab2ec5d5375651ff3e70ce00e"} Sep 19 11:01:44.359293 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:44.359194 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" Sep 19 11:01:44.374117 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:44.374042 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" podStartSLOduration=1.374027794 podStartE2EDuration="1.374027794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:01:44.371579926 +0000 UTC m=+1636.284084645" watchObservedRunningTime="2026-09-19 11:01:44.374027794 +0000 UTC m=+1636.286532526" Sep 19 11:01:44.385975 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:44.385910 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" podStartSLOduration=1.385900369 podStartE2EDuration="1.385900369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:01:44.385291546 +0000 UTC m=+1636.297796268" watchObservedRunningTime="2026-09-19 11:01:44.385900369 +0000 UTC m=+1636.298405086" Sep 19 11:01:55.367677 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:55.367647 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" Sep 19 11:01:55.368255 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:01:55.367712 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" Sep 19 11:02:11.036977 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:11.036950 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j_8fe20919-b1cb-4568-bd6c-381f584f55db/payload-processing/0.log" Sep 19 11:02:30.557695 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:30.557661 2579 scope.go:117] "RemoveContainer" containerID="ffd59e3e04b5a55655694965b329ea46b142ecd440547212e3fe3df935f4d2f9" Sep 19 11:02:30.566516 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:30.566495 2579 scope.go:117] "RemoveContainer" containerID="af60fc7cb2b3d93796eed5e735db6032a9d451b56d95645ecf25a01f416c09d7" Sep 19 11:02:39.210475 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:39.210446 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j_8fe20919-b1cb-4568-bd6c-381f584f55db/payload-processing/0.log" Sep 19 11:02:46.515785 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:46.515699 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj"] Sep 19 11:02:46.521043 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:46.521025 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" Sep 19 11:02:46.523520 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:46.523498 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-953b43\"" Sep 19 11:02:46.523652 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:46.523592 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-953b43-dockercfg-shtdj\"" Sep 19 11:02:46.530251 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:46.530228 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj"] Sep 19 11:02:46.621307 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:46.621145 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5dc\" (UniqueName: \"kubernetes.io/projected/1147152e-0d56-4d17-82b9-3ee2111a1cde-kube-api-access-rf5dc\") pod \"payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj\" (UID: \"1147152e-0d56-4d17-82b9-3ee2111a1cde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" Sep 19 11:02:46.621307 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:46.621220 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1147152e-0d56-4d17-82b9-3ee2111a1cde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj\" (UID: \"1147152e-0d56-4d17-82b9-3ee2111a1cde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" Sep 19 11:02:46.722096 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:46.722042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5dc\" (UniqueName: \"kubernetes.io/projected/1147152e-0d56-4d17-82b9-3ee2111a1cde-kube-api-access-rf5dc\") pod \"payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj\" (UID: \"1147152e-0d56-4d17-82b9-3ee2111a1cde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" Sep 19 11:02:46.722282 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:46.722110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1147152e-0d56-4d17-82b9-3ee2111a1cde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj\" (UID: \"1147152e-0d56-4d17-82b9-3ee2111a1cde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" Sep 19 11:02:46.722837 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:46.722813 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1147152e-0d56-4d17-82b9-3ee2111a1cde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj\" (UID: \"1147152e-0d56-4d17-82b9-3ee2111a1cde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" Sep 19 11:02:46.731108 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:46.731081 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5dc\" (UniqueName: \"kubernetes.io/projected/1147152e-0d56-4d17-82b9-3ee2111a1cde-kube-api-access-rf5dc\") pod \"payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj\" (UID: \"1147152e-0d56-4d17-82b9-3ee2111a1cde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" Sep 19 11:02:46.833103 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:46.833016 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" Sep 19 11:02:47.103216 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:47.103191 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj"] Sep 19 11:02:47.105608 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:02:47.105577 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1147152e_0d56_4d17_82b9_3ee2111a1cde.slice/crio-f75b8ba976fa1efe45d2b16e4926e62c5b419940c32afbb2c29f157008bfd719 WatchSource:0}: Error finding container f75b8ba976fa1efe45d2b16e4926e62c5b419940c32afbb2c29f157008bfd719: Status 404 returned error can't find the container with id f75b8ba976fa1efe45d2b16e4926e62c5b419940c32afbb2c29f157008bfd719 Sep 19 11:02:47.720952 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:47.720916 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" event={"ID":"1147152e-0d56-4d17-82b9-3ee2111a1cde","Type":"ContainerStarted","Data":"518638ae7ca43501dea3847738b2dc70b42765040ecb2701093aef076cd0751e"} Sep 19 11:02:47.721355 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:47.720971 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" event={"ID":"1147152e-0d56-4d17-82b9-3ee2111a1cde","Type":"ContainerStarted","Data":"f75b8ba976fa1efe45d2b16e4926e62c5b419940c32afbb2c29f157008bfd719"} Sep 19 11:02:47.721355 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:47.721049 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" Sep 19 11:02:47.735378 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:47.735329 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" podStartSLOduration=1.735312688 podStartE2EDuration="1.735312688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:02:47.734614572 +0000 UTC m=+1699.647119277" watchObservedRunningTime="2026-09-19 11:02:47.735312688 +0000 UTC m=+1699.647817404" Sep 19 11:02:58.729186 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:02:58.729156 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" Sep 19 11:03:08.840017 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:08.839985 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642"] Sep 19 11:03:08.843705 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:08.843689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642" Sep 19 11:03:08.846144 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:08.846123 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 11:03:08.846289 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:08.846123 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 11:03:08.850592 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:08.850569 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642"] Sep 19 11:03:08.931668 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:08.931629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/386c801f-4dbf-4c09-ba72-30aee4dab29f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642\" (UID: \"386c801f-4dbf-4c09-ba72-30aee4dab29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642" Sep 19 11:03:09.032943 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:09.032904 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/386c801f-4dbf-4c09-ba72-30aee4dab29f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642\" (UID: \"386c801f-4dbf-4c09-ba72-30aee4dab29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642" Sep 19 11:03:09.033567 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:09.033545 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/386c801f-4dbf-4c09-ba72-30aee4dab29f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642\" (UID: \"386c801f-4dbf-4c09-ba72-30aee4dab29f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642" Sep 19 11:03:09.156181 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:09.156157 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642" Sep 19 11:03:09.425280 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:09.425254 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642"] Sep 19 11:03:09.427037 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:03:09.427008 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386c801f_4dbf_4c09_ba72_30aee4dab29f.slice/crio-82e97db518c71d056e44971982f374817893d3ff0858b2bd2e16c7b8c70a6153 WatchSource:0}: Error finding container 82e97db518c71d056e44971982f374817893d3ff0858b2bd2e16c7b8c70a6153: Status 404 returned error can't find the container with id 82e97db518c71d056e44971982f374817893d3ff0858b2bd2e16c7b8c70a6153 Sep 19 11:03:09.846768 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:09.846687 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642" event={"ID":"386c801f-4dbf-4c09-ba72-30aee4dab29f","Type":"ContainerStarted","Data":"b12edc58514626f33bfe918b8c9a29d3c387862ae1780a89207f01ecd8f1fcec"} Sep 19 11:03:09.846768 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:09.846737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642" event={"ID":"386c801f-4dbf-4c09-ba72-30aee4dab29f","Type":"ContainerStarted","Data":"82e97db518c71d056e44971982f374817893d3ff0858b2bd2e16c7b8c70a6153"} Sep 19 11:03:09.861866 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:09.861825 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642" podStartSLOduration=1.861811122 podStartE2EDuration="1.861811122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:03:09.859958458 +0000 UTC m=+1721.772463175" watchObservedRunningTime="2026-09-19 11:03:09.861811122 +0000 UTC m=+1721.774315837" Sep 19 11:03:10.853798 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:10.853761 2579 generic.go:356] "Generic (PLEG): container finished" podID="386c801f-4dbf-4c09-ba72-30aee4dab29f" containerID="b12edc58514626f33bfe918b8c9a29d3c387862ae1780a89207f01ecd8f1fcec" exitCode=0 Sep 19 11:03:10.854192 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:10.853845 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642" event={"ID":"386c801f-4dbf-4c09-ba72-30aee4dab29f","Type":"ContainerDied","Data":"b12edc58514626f33bfe918b8c9a29d3c387862ae1780a89207f01ecd8f1fcec"} Sep 19 11:03:11.995148 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:11.995107 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642" Sep 19 11:03:12.061574 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:12.061540 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/386c801f-4dbf-4c09-ba72-30aee4dab29f-maas-api-service-ca\") pod \"386c801f-4dbf-4c09-ba72-30aee4dab29f\" (UID: \"386c801f-4dbf-4c09-ba72-30aee4dab29f\") " Sep 19 11:03:12.061841 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:12.061820 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386c801f-4dbf-4c09-ba72-30aee4dab29f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "386c801f-4dbf-4c09-ba72-30aee4dab29f" (UID: "386c801f-4dbf-4c09-ba72-30aee4dab29f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:03:12.061926 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:12.061907 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/386c801f-4dbf-4c09-ba72-30aee4dab29f-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:03:12.866741 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:12.866712 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642" Sep 19 11:03:12.866741 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:12.866730 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642" event={"ID":"386c801f-4dbf-4c09-ba72-30aee4dab29f","Type":"ContainerDied","Data":"82e97db518c71d056e44971982f374817893d3ff0858b2bd2e16c7b8c70a6153"} Sep 19 11:03:12.866955 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:12.866798 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82e97db518c71d056e44971982f374817893d3ff0858b2bd2e16c7b8c70a6153" Sep 19 11:03:15.018945 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.018910 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj"] Sep 19 11:03:15.019420 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.019176 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" podUID="1147152e-0d56-4d17-82b9-3ee2111a1cde" containerName="payload-pre-processing" containerID="cri-o://518638ae7ca43501dea3847738b2dc70b42765040ecb2701093aef076cd0751e" gracePeriod=30 Sep 19 11:03:15.152793 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.152749 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642"] Sep 19 11:03:15.155685 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.155655 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-953b43-xw642"] Sep 19 11:03:15.294156 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.294132 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" Sep 19 11:03:15.389837 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.389808 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf5dc\" (UniqueName: \"kubernetes.io/projected/1147152e-0d56-4d17-82b9-3ee2111a1cde-kube-api-access-rf5dc\") pod \"1147152e-0d56-4d17-82b9-3ee2111a1cde\" (UID: \"1147152e-0d56-4d17-82b9-3ee2111a1cde\") " Sep 19 11:03:15.389996 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.389896 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1147152e-0d56-4d17-82b9-3ee2111a1cde-plugins-config-volume\") pod \"1147152e-0d56-4d17-82b9-3ee2111a1cde\" (UID: \"1147152e-0d56-4d17-82b9-3ee2111a1cde\") " Sep 19 11:03:15.390263 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.390239 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1147152e-0d56-4d17-82b9-3ee2111a1cde-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1147152e-0d56-4d17-82b9-3ee2111a1cde" (UID: "1147152e-0d56-4d17-82b9-3ee2111a1cde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:03:15.391917 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.391896 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1147152e-0d56-4d17-82b9-3ee2111a1cde-kube-api-access-rf5dc" (OuterVolumeSpecName: "kube-api-access-rf5dc") pod "1147152e-0d56-4d17-82b9-3ee2111a1cde" (UID: "1147152e-0d56-4d17-82b9-3ee2111a1cde"). InnerVolumeSpecName "kube-api-access-rf5dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:03:15.491462 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.491427 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf5dc\" (UniqueName: \"kubernetes.io/projected/1147152e-0d56-4d17-82b9-3ee2111a1cde-kube-api-access-rf5dc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:03:15.491618 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.491490 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1147152e-0d56-4d17-82b9-3ee2111a1cde-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:03:15.887383 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.887346 2579 generic.go:356] "Generic (PLEG): container finished" podID="1147152e-0d56-4d17-82b9-3ee2111a1cde" containerID="518638ae7ca43501dea3847738b2dc70b42765040ecb2701093aef076cd0751e" exitCode=0 Sep 19 11:03:15.887545 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.887392 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" Sep 19 11:03:15.887545 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.887421 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" event={"ID":"1147152e-0d56-4d17-82b9-3ee2111a1cde","Type":"ContainerDied","Data":"518638ae7ca43501dea3847738b2dc70b42765040ecb2701093aef076cd0751e"} Sep 19 11:03:15.887545 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.887476 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj" event={"ID":"1147152e-0d56-4d17-82b9-3ee2111a1cde","Type":"ContainerDied","Data":"f75b8ba976fa1efe45d2b16e4926e62c5b419940c32afbb2c29f157008bfd719"} Sep 19 11:03:15.887545 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.887499 2579 scope.go:117] "RemoveContainer" containerID="518638ae7ca43501dea3847738b2dc70b42765040ecb2701093aef076cd0751e" Sep 19 11:03:15.896886 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.896871 2579 scope.go:117] "RemoveContainer" containerID="518638ae7ca43501dea3847738b2dc70b42765040ecb2701093aef076cd0751e" Sep 19 11:03:15.897305 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:03:15.897280 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518638ae7ca43501dea3847738b2dc70b42765040ecb2701093aef076cd0751e\": container with ID starting with 518638ae7ca43501dea3847738b2dc70b42765040ecb2701093aef076cd0751e not found: ID does not exist" containerID="518638ae7ca43501dea3847738b2dc70b42765040ecb2701093aef076cd0751e" Sep 19 11:03:15.897413 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.897315 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518638ae7ca43501dea3847738b2dc70b42765040ecb2701093aef076cd0751e"} err="failed to get container status \"518638ae7ca43501dea3847738b2dc70b42765040ecb2701093aef076cd0751e\": rpc error: code = NotFound desc = could not find container \"518638ae7ca43501dea3847738b2dc70b42765040ecb2701093aef076cd0751e\": container with ID starting with 518638ae7ca43501dea3847738b2dc70b42765040ecb2701093aef076cd0751e not found: ID does not exist" Sep 19 11:03:15.910977 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.910953 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj"] Sep 19 11:03:15.912999 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:15.912979 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj"] Sep 19 11:03:16.196142 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:16.196045 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1147152e-0d56-4d17-82b9-3ee2111a1cde" path="/var/lib/kubelet/pods/1147152e-0d56-4d17-82b9-3ee2111a1cde/volumes" Sep 19 11:03:16.196493 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:16.196413 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="386c801f-4dbf-4c09-ba72-30aee4dab29f" path="/var/lib/kubelet/pods/386c801f-4dbf-4c09-ba72-30aee4dab29f/volumes" Sep 19 11:03:24.965211 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:24.965175 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf"] Sep 19 11:03:24.965600 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:24.965574 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1147152e-0d56-4d17-82b9-3ee2111a1cde" containerName="payload-pre-processing" Sep 19 11:03:24.965600 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:24.965585 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1147152e-0d56-4d17-82b9-3ee2111a1cde" containerName="payload-pre-processing" Sep 19 11:03:24.965672 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:24.965607 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386c801f-4dbf-4c09-ba72-30aee4dab29f" containerName="revoke-keys" Sep 19 11:03:24.965672 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:24.965613 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="386c801f-4dbf-4c09-ba72-30aee4dab29f" containerName="revoke-keys" Sep 19 11:03:24.965672 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:24.965668 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="386c801f-4dbf-4c09-ba72-30aee4dab29f" containerName="revoke-keys" Sep 19 11:03:24.965770 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:24.965677 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1147152e-0d56-4d17-82b9-3ee2111a1cde" containerName="payload-pre-processing" Sep 19 11:03:24.968558 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:24.968542 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf" Sep 19 11:03:24.971253 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:24.971229 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 11:03:24.971628 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:24.971609 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 11:03:24.994072 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:24.994041 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf"] Sep 19 11:03:25.083191 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:25.083162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27acf61-8a10-4d84-8a08-d9a52acc333c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf\" (UID: \"a27acf61-8a10-4d84-8a08-d9a52acc333c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf" Sep 19 11:03:25.184303 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:25.184270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27acf61-8a10-4d84-8a08-d9a52acc333c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf\" (UID: \"a27acf61-8a10-4d84-8a08-d9a52acc333c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf" Sep 19 11:03:25.184878 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:25.184861 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27acf61-8a10-4d84-8a08-d9a52acc333c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf\" (UID: \"a27acf61-8a10-4d84-8a08-d9a52acc333c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf" Sep 19 11:03:25.279140 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:25.279040 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf" Sep 19 11:03:25.547539 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:25.547515 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf"] Sep 19 11:03:25.549572 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:03:25.549545 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27acf61_8a10_4d84_8a08_d9a52acc333c.slice/crio-a5279011047e7a0786356c9763f35ae98a7a29cdf421bebd50688f9d69e981b6 WatchSource:0}: Error finding container a5279011047e7a0786356c9763f35ae98a7a29cdf421bebd50688f9d69e981b6: Status 404 returned error can't find the container with id a5279011047e7a0786356c9763f35ae98a7a29cdf421bebd50688f9d69e981b6 Sep 19 11:03:25.948372 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:25.948342 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf" event={"ID":"a27acf61-8a10-4d84-8a08-d9a52acc333c","Type":"ContainerStarted","Data":"747295ff7b2dfcb3b6dfc0c2dc0af8c714718b7a99415d3631f0e35b119bee41"} Sep 19 11:03:25.948542 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:25.948391 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf" event={"ID":"a27acf61-8a10-4d84-8a08-d9a52acc333c","Type":"ContainerStarted","Data":"a5279011047e7a0786356c9763f35ae98a7a29cdf421bebd50688f9d69e981b6"} Sep 19 11:03:25.968353 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:25.968300 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf" podStartSLOduration=1.968286511 podStartE2EDuration="1.968286511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:03:25.966102068 +0000 UTC m=+1737.878606798" watchObservedRunningTime="2026-09-19 11:03:25.968286511 +0000 UTC m=+1737.880791281" Sep 19 11:03:26.955176 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:26.955141 2579 generic.go:356] "Generic (PLEG): container finished" podID="a27acf61-8a10-4d84-8a08-d9a52acc333c" containerID="747295ff7b2dfcb3b6dfc0c2dc0af8c714718b7a99415d3631f0e35b119bee41" exitCode=0 Sep 19 11:03:26.955369 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:26.955231 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf" event={"ID":"a27acf61-8a10-4d84-8a08-d9a52acc333c","Type":"ContainerDied","Data":"747295ff7b2dfcb3b6dfc0c2dc0af8c714718b7a99415d3631f0e35b119bee41"} Sep 19 11:03:28.095829 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:28.095806 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf" Sep 19 11:03:28.211793 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:28.211721 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27acf61-8a10-4d84-8a08-d9a52acc333c-maas-api-service-ca\") pod \"a27acf61-8a10-4d84-8a08-d9a52acc333c\" (UID: \"a27acf61-8a10-4d84-8a08-d9a52acc333c\") " Sep 19 11:03:28.212052 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:28.212030 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27acf61-8a10-4d84-8a08-d9a52acc333c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a27acf61-8a10-4d84-8a08-d9a52acc333c" (UID: "a27acf61-8a10-4d84-8a08-d9a52acc333c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:03:28.312806 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:28.312781 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27acf61-8a10-4d84-8a08-d9a52acc333c-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:03:28.967473 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:28.967443 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf" Sep 19 11:03:28.967672 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:28.967439 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf" event={"ID":"a27acf61-8a10-4d84-8a08-d9a52acc333c","Type":"ContainerDied","Data":"a5279011047e7a0786356c9763f35ae98a7a29cdf421bebd50688f9d69e981b6"} Sep 19 11:03:28.967672 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:28.967564 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5279011047e7a0786356c9763f35ae98a7a29cdf421bebd50688f9d69e981b6" Sep 19 11:03:40.552848 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:40.552808 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf"] Sep 19 11:03:40.554734 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:40.554710 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd3aad-gg8lf"] Sep 19 11:03:42.196744 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:03:42.196711 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a27acf61-8a10-4d84-8a08-d9a52acc333c" path="/var/lib/kubelet/pods/a27acf61-8a10-4d84-8a08-d9a52acc333c/volumes" Sep 19 11:04:29.414942 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:29.414907 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh"] Sep 19 11:04:29.415464 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:29.415446 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27acf61-8a10-4d84-8a08-d9a52acc333c" containerName="revoke-keys" Sep 19 11:04:29.415511 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:29.415466 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27acf61-8a10-4d84-8a08-d9a52acc333c" containerName="revoke-keys" Sep 19 11:04:29.415553 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:29.415544 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27acf61-8a10-4d84-8a08-d9a52acc333c" containerName="revoke-keys" Sep 19 11:04:29.418613 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:29.418592 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh" Sep 19 11:04:29.420891 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:29.420871 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 11:04:29.421070 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:29.421038 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 11:04:29.425027 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:29.425004 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh"] Sep 19 11:04:29.446544 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:29.446523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1016e101-91e9-4245-b8d7-993c0534f885-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh\" (UID: \"1016e101-91e9-4245-b8d7-993c0534f885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh" Sep 19 11:04:29.547871 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:29.547844 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1016e101-91e9-4245-b8d7-993c0534f885-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh\" (UID: \"1016e101-91e9-4245-b8d7-993c0534f885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh" Sep 19 11:04:29.548483 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:29.548460 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1016e101-91e9-4245-b8d7-993c0534f885-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh\" (UID: \"1016e101-91e9-4245-b8d7-993c0534f885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh" Sep 19 11:04:29.731360 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:29.731277 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh" Sep 19 11:04:30.003085 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:30.003047 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh"] Sep 19 11:04:30.312609 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:30.312525 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh" event={"ID":"1016e101-91e9-4245-b8d7-993c0534f885","Type":"ContainerStarted","Data":"349d5f5e1458c2fc05718b5c48b124e1c48f9a9e778137c5c009c90143d932e6"} Sep 19 11:04:30.329785 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:30.329760 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 11:04:30.331930 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:30.331910 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 11:04:31.318956 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:31.318921 2579 generic.go:356] "Generic (PLEG): container finished" podID="1016e101-91e9-4245-b8d7-993c0534f885" containerID="99959df5f65ced36f97a2ca9a445fffcf351e226b072637ea1cafa0d70579ef8" exitCode=0 Sep 19 11:04:31.319350 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:31.319008 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh" event={"ID":"1016e101-91e9-4245-b8d7-993c0534f885","Type":"ContainerDied","Data":"99959df5f65ced36f97a2ca9a445fffcf351e226b072637ea1cafa0d70579ef8"} Sep 19 11:04:32.463459 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:32.463429 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh" Sep 19 11:04:32.576267 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:32.576235 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1016e101-91e9-4245-b8d7-993c0534f885-maas-api-service-ca\") pod \"1016e101-91e9-4245-b8d7-993c0534f885\" (UID: \"1016e101-91e9-4245-b8d7-993c0534f885\") " Sep 19 11:04:32.576553 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:32.576530 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1016e101-91e9-4245-b8d7-993c0534f885-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1016e101-91e9-4245-b8d7-993c0534f885" (UID: "1016e101-91e9-4245-b8d7-993c0534f885"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:04:32.677331 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:32.677302 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1016e101-91e9-4245-b8d7-993c0534f885-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:04:33.331763 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:33.331736 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh" Sep 19 11:04:33.331763 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:33.331744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh" event={"ID":"1016e101-91e9-4245-b8d7-993c0534f885","Type":"ContainerDied","Data":"349d5f5e1458c2fc05718b5c48b124e1c48f9a9e778137c5c009c90143d932e6"} Sep 19 11:04:33.331972 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:33.331783 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="349d5f5e1458c2fc05718b5c48b124e1c48f9a9e778137c5c009c90143d932e6" Sep 19 11:04:39.921724 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:39.921684 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j"] Sep 19 11:04:39.922198 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:39.922000 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" podUID="8fe20919-b1cb-4568-bd6c-381f584f55db" containerName="payload-processing" containerID="cri-o://ca2d92413e960591baea15215e75daec154f961e9c8613a01c0aa3d075f232fa" gracePeriod=30 Sep 19 11:04:39.927330 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:39.927300 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn"] Sep 19 11:04:39.927600 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:39.927570 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" podUID="01b78706-202f-439a-826a-ba9b2db6a55c" containerName="payload-pre-processing" containerID="cri-o://91dca0bd89dae46978708cb449081e206601ef0703f05b2cd95a2bebbfacddba" gracePeriod=30 Sep 19 11:04:40.082027 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.081984 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh"] Sep 19 11:04:40.088364 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.088323 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f210d9-chsrh"] Sep 19 11:04:40.201297 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.201270 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1016e101-91e9-4245-b8d7-993c0534f885" path="/var/lib/kubelet/pods/1016e101-91e9-4245-b8d7-993c0534f885/volumes" Sep 19 11:04:40.215699 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.215678 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" Sep 19 11:04:40.234244 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.234222 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" Sep 19 11:04:40.256728 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.256572 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jfhq\" (UniqueName: \"kubernetes.io/projected/01b78706-202f-439a-826a-ba9b2db6a55c-kube-api-access-6jfhq\") pod \"01b78706-202f-439a-826a-ba9b2db6a55c\" (UID: \"01b78706-202f-439a-826a-ba9b2db6a55c\") " Sep 19 11:04:40.256728 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.256621 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b78706-202f-439a-826a-ba9b2db6a55c-plugins-config-volume\") pod \"01b78706-202f-439a-826a-ba9b2db6a55c\" (UID: \"01b78706-202f-439a-826a-ba9b2db6a55c\") " Sep 19 11:04:40.257995 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.257950 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b78706-202f-439a-826a-ba9b2db6a55c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01b78706-202f-439a-826a-ba9b2db6a55c" (UID: "01b78706-202f-439a-826a-ba9b2db6a55c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:04:40.260808 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.260781 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b78706-202f-439a-826a-ba9b2db6a55c-kube-api-access-6jfhq" (OuterVolumeSpecName: "kube-api-access-6jfhq") pod "01b78706-202f-439a-826a-ba9b2db6a55c" (UID: "01b78706-202f-439a-826a-ba9b2db6a55c"). InnerVolumeSpecName "kube-api-access-6jfhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:04:40.357558 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.357520 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk7lc\" (UniqueName: \"kubernetes.io/projected/8fe20919-b1cb-4568-bd6c-381f584f55db-kube-api-access-wk7lc\") pod \"8fe20919-b1cb-4568-bd6c-381f584f55db\" (UID: \"8fe20919-b1cb-4568-bd6c-381f584f55db\") " Sep 19 11:04:40.357740 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.357576 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe20919-b1cb-4568-bd6c-381f584f55db-plugins-config-volume\") pod \"8fe20919-b1cb-4568-bd6c-381f584f55db\" (UID: \"8fe20919-b1cb-4568-bd6c-381f584f55db\") " Sep 19 11:04:40.357922 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.357904 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jfhq\" (UniqueName: \"kubernetes.io/projected/01b78706-202f-439a-826a-ba9b2db6a55c-kube-api-access-6jfhq\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:04:40.357992 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.357926 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b78706-202f-439a-826a-ba9b2db6a55c-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:04:40.357992 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.357923 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe20919-b1cb-4568-bd6c-381f584f55db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8fe20919-b1cb-4568-bd6c-381f584f55db" (UID: "8fe20919-b1cb-4568-bd6c-381f584f55db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:04:40.359815 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.359790 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe20919-b1cb-4568-bd6c-381f584f55db-kube-api-access-wk7lc" (OuterVolumeSpecName: "kube-api-access-wk7lc") pod "8fe20919-b1cb-4568-bd6c-381f584f55db" (UID: "8fe20919-b1cb-4568-bd6c-381f584f55db"). InnerVolumeSpecName "kube-api-access-wk7lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:04:40.375320 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.375289 2579 generic.go:356] "Generic (PLEG): container finished" podID="8fe20919-b1cb-4568-bd6c-381f584f55db" containerID="ca2d92413e960591baea15215e75daec154f961e9c8613a01c0aa3d075f232fa" exitCode=0 Sep 19 11:04:40.375460 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.375342 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" Sep 19 11:04:40.375460 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.375378 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" event={"ID":"8fe20919-b1cb-4568-bd6c-381f584f55db","Type":"ContainerDied","Data":"ca2d92413e960591baea15215e75daec154f961e9c8613a01c0aa3d075f232fa"} Sep 19 11:04:40.375460 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.375435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j" event={"ID":"8fe20919-b1cb-4568-bd6c-381f584f55db","Type":"ContainerDied","Data":"59057ca6aa4971819a906bd481f3e20a6d4e455ab2ec5d5375651ff3e70ce00e"} Sep 19 11:04:40.375640 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.375464 2579 scope.go:117] "RemoveContainer" containerID="ca2d92413e960591baea15215e75daec154f961e9c8613a01c0aa3d075f232fa" Sep 19 11:04:40.376993 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.376968 2579 generic.go:356] "Generic (PLEG): container finished" podID="01b78706-202f-439a-826a-ba9b2db6a55c" containerID="91dca0bd89dae46978708cb449081e206601ef0703f05b2cd95a2bebbfacddba" exitCode=0 Sep 19 11:04:40.377195 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.377002 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" event={"ID":"01b78706-202f-439a-826a-ba9b2db6a55c","Type":"ContainerDied","Data":"91dca0bd89dae46978708cb449081e206601ef0703f05b2cd95a2bebbfacddba"} Sep 19 11:04:40.377195 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.377019 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" Sep 19 11:04:40.377195 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.377071 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn" event={"ID":"01b78706-202f-439a-826a-ba9b2db6a55c","Type":"ContainerDied","Data":"48781bee1e23e140b87755a6622fc4c9f5c610f57f49954b30773e8f3e799250"} Sep 19 11:04:40.386344 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.386301 2579 scope.go:117] "RemoveContainer" containerID="ca2d92413e960591baea15215e75daec154f961e9c8613a01c0aa3d075f232fa" Sep 19 11:04:40.386642 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:04:40.386619 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca2d92413e960591baea15215e75daec154f961e9c8613a01c0aa3d075f232fa\": container with ID starting with ca2d92413e960591baea15215e75daec154f961e9c8613a01c0aa3d075f232fa not found: ID does not exist" containerID="ca2d92413e960591baea15215e75daec154f961e9c8613a01c0aa3d075f232fa" Sep 19 11:04:40.386720 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.386653 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2d92413e960591baea15215e75daec154f961e9c8613a01c0aa3d075f232fa"} err="failed to get container status \"ca2d92413e960591baea15215e75daec154f961e9c8613a01c0aa3d075f232fa\": rpc error: code = NotFound desc = could not find container \"ca2d92413e960591baea15215e75daec154f961e9c8613a01c0aa3d075f232fa\": container with ID starting with ca2d92413e960591baea15215e75daec154f961e9c8613a01c0aa3d075f232fa not found: ID does not exist" Sep 19 11:04:40.386720 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.386674 2579 scope.go:117] "RemoveContainer" containerID="91dca0bd89dae46978708cb449081e206601ef0703f05b2cd95a2bebbfacddba" Sep 19 11:04:40.398933 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.398912 2579 scope.go:117] "RemoveContainer" containerID="91dca0bd89dae46978708cb449081e206601ef0703f05b2cd95a2bebbfacddba" Sep 19 11:04:40.399536 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:04:40.399509 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91dca0bd89dae46978708cb449081e206601ef0703f05b2cd95a2bebbfacddba\": container with ID starting with 91dca0bd89dae46978708cb449081e206601ef0703f05b2cd95a2bebbfacddba not found: ID does not exist" containerID="91dca0bd89dae46978708cb449081e206601ef0703f05b2cd95a2bebbfacddba" Sep 19 11:04:40.399614 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.399543 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91dca0bd89dae46978708cb449081e206601ef0703f05b2cd95a2bebbfacddba"} err="failed to get container status \"91dca0bd89dae46978708cb449081e206601ef0703f05b2cd95a2bebbfacddba\": rpc error: code = NotFound desc = could not find container \"91dca0bd89dae46978708cb449081e206601ef0703f05b2cd95a2bebbfacddba\": container with ID starting with 91dca0bd89dae46978708cb449081e206601ef0703f05b2cd95a2bebbfacddba not found: ID does not exist" Sep 19 11:04:40.401540 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.401519 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j"] Sep 19 11:04:40.404577 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.404556 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j"] Sep 19 11:04:40.415589 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.415568 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn"] Sep 19 11:04:40.417432 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.417413 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn"] Sep 19 11:04:40.458710 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.458641 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk7lc\" (UniqueName: \"kubernetes.io/projected/8fe20919-b1cb-4568-bd6c-381f584f55db-kube-api-access-wk7lc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:04:40.458710 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:40.458666 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe20919-b1cb-4568-bd6c-381f584f55db-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:04:42.196405 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:42.196373 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b78706-202f-439a-826a-ba9b2db6a55c" path="/var/lib/kubelet/pods/01b78706-202f-439a-826a-ba9b2db6a55c/volumes" Sep 19 11:04:42.196762 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:42.196734 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fe20919-b1cb-4568-bd6c-381f584f55db" path="/var/lib/kubelet/pods/8fe20919-b1cb-4568-bd6c-381f584f55db/volumes" Sep 19 11:04:46.871801 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.871764 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn"] Sep 19 11:04:46.872251 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.872231 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1016e101-91e9-4245-b8d7-993c0534f885" containerName="revoke-keys" Sep 19 11:04:46.872315 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.872254 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1016e101-91e9-4245-b8d7-993c0534f885" containerName="revoke-keys" Sep 19 11:04:46.872315 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.872276 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe20919-b1cb-4568-bd6c-381f584f55db" containerName="payload-processing" Sep 19 11:04:46.872315 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.872282 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe20919-b1cb-4568-bd6c-381f584f55db" containerName="payload-processing" Sep 19 11:04:46.872315 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.872290 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b78706-202f-439a-826a-ba9b2db6a55c" containerName="payload-pre-processing" Sep 19 11:04:46.872315 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.872296 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b78706-202f-439a-826a-ba9b2db6a55c" containerName="payload-pre-processing" Sep 19 11:04:46.872475 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.872363 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b78706-202f-439a-826a-ba9b2db6a55c" containerName="payload-pre-processing" Sep 19 11:04:46.872475 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.872373 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe20919-b1cb-4568-bd6c-381f584f55db" containerName="payload-processing" Sep 19 11:04:46.872475 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.872388 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1016e101-91e9-4245-b8d7-993c0534f885" containerName="revoke-keys" Sep 19 11:04:46.877220 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.877205 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn" Sep 19 11:04:46.879918 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.879892 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 11:04:46.880096 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.879893 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 11:04:46.881437 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:46.881409 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn"] Sep 19 11:04:47.016323 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:47.016292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c80a1c-e9e2-4556-80c1-75fcc6c1913f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn\" (UID: \"b8c80a1c-e9e2-4556-80c1-75fcc6c1913f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn" Sep 19 11:04:47.117626 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:47.117595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c80a1c-e9e2-4556-80c1-75fcc6c1913f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn\" (UID: \"b8c80a1c-e9e2-4556-80c1-75fcc6c1913f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn" Sep 19 11:04:47.118251 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:47.118234 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c80a1c-e9e2-4556-80c1-75fcc6c1913f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn\" (UID: \"b8c80a1c-e9e2-4556-80c1-75fcc6c1913f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn" Sep 19 11:04:47.189882 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:47.189847 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn" Sep 19 11:04:47.465884 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:47.465857 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn"] Sep 19 11:04:47.467803 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:04:47.467757 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c80a1c_e9e2_4556_80c1_75fcc6c1913f.slice/crio-c445e0a6aa51d47b10a37405c95cabf4027504500794a3a8fa0c4ee84be7c9ca WatchSource:0}: Error finding container c445e0a6aa51d47b10a37405c95cabf4027504500794a3a8fa0c4ee84be7c9ca: Status 404 returned error can't find the container with id c445e0a6aa51d47b10a37405c95cabf4027504500794a3a8fa0c4ee84be7c9ca Sep 19 11:04:48.425514 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:48.425424 2579 generic.go:356] "Generic (PLEG): container finished" podID="b8c80a1c-e9e2-4556-80c1-75fcc6c1913f" containerID="317ffb8ab3a43c8508e727707c09ae93d3694753a3898dfcb7fbd383f3e3428b" exitCode=0 Sep 19 11:04:48.425884 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:48.425517 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn" event={"ID":"b8c80a1c-e9e2-4556-80c1-75fcc6c1913f","Type":"ContainerDied","Data":"317ffb8ab3a43c8508e727707c09ae93d3694753a3898dfcb7fbd383f3e3428b"} Sep 19 11:04:48.425884 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:48.425581 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn" event={"ID":"b8c80a1c-e9e2-4556-80c1-75fcc6c1913f","Type":"ContainerStarted","Data":"c445e0a6aa51d47b10a37405c95cabf4027504500794a3a8fa0c4ee84be7c9ca"} Sep 19 11:04:49.566136 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:49.566112 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn" Sep 19 11:04:49.641173 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:49.641145 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c80a1c-e9e2-4556-80c1-75fcc6c1913f-maas-api-service-ca\") pod \"b8c80a1c-e9e2-4556-80c1-75fcc6c1913f\" (UID: \"b8c80a1c-e9e2-4556-80c1-75fcc6c1913f\") " Sep 19 11:04:49.641539 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:49.641516 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c80a1c-e9e2-4556-80c1-75fcc6c1913f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b8c80a1c-e9e2-4556-80c1-75fcc6c1913f" (UID: "b8c80a1c-e9e2-4556-80c1-75fcc6c1913f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:04:49.742189 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:49.742114 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c80a1c-e9e2-4556-80c1-75fcc6c1913f-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:04:50.438324 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:50.438295 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn" Sep 19 11:04:50.438324 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:50.438311 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn" event={"ID":"b8c80a1c-e9e2-4556-80c1-75fcc6c1913f","Type":"ContainerDied","Data":"c445e0a6aa51d47b10a37405c95cabf4027504500794a3a8fa0c4ee84be7c9ca"} Sep 19 11:04:50.438572 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:50.438364 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c445e0a6aa51d47b10a37405c95cabf4027504500794a3a8fa0c4ee84be7c9ca" Sep 19 11:04:57.317495 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.317456 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t"] Sep 19 11:04:57.319387 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.319364 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-72e262-29830260-jvl5t"] Sep 19 11:04:57.340447 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.340418 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb"] Sep 19 11:04:57.340688 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.340656 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" podUID="46ede73c-f5b1-4625-8073-fc5401026a40" containerName="payload-processing" containerID="cri-o://071de16aa68c6ac8ca62d4f93c28e24639683d434290acb361f7d0df07b0313e" gracePeriod=30 Sep 19 11:04:57.346439 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.346411 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28"] Sep 19 11:04:57.346626 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.346605 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" podUID="b9af77da-8c73-477a-9db8-1f12e966f951" containerName="payload-pre-processing" containerID="cri-o://a08c561eda8273dbc9a30f074f6b809a76f914ec81987b2e1193745a36ad349c" gracePeriod=30 Sep 19 11:04:57.480011 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.479977 2579 generic.go:356] "Generic (PLEG): container finished" podID="b9af77da-8c73-477a-9db8-1f12e966f951" containerID="a08c561eda8273dbc9a30f074f6b809a76f914ec81987b2e1193745a36ad349c" exitCode=0 Sep 19 11:04:57.480183 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.480080 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" event={"ID":"b9af77da-8c73-477a-9db8-1f12e966f951","Type":"ContainerDied","Data":"a08c561eda8273dbc9a30f074f6b809a76f914ec81987b2e1193745a36ad349c"} Sep 19 11:04:57.482177 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.482137 2579 generic.go:356] "Generic (PLEG): container finished" podID="46ede73c-f5b1-4625-8073-fc5401026a40" containerID="071de16aa68c6ac8ca62d4f93c28e24639683d434290acb361f7d0df07b0313e" exitCode=0 Sep 19 11:04:57.482294 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.482182 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" event={"ID":"46ede73c-f5b1-4625-8073-fc5401026a40","Type":"ContainerDied","Data":"071de16aa68c6ac8ca62d4f93c28e24639683d434290acb361f7d0df07b0313e"} Sep 19 11:04:57.494290 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.494254 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn"] Sep 19 11:04:57.496641 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.496609 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-72e262-kpjpn"] Sep 19 11:04:57.601923 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.601896 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" Sep 19 11:04:57.629714 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.629690 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" Sep 19 11:04:57.715507 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.715472 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g7tp\" (UniqueName: \"kubernetes.io/projected/46ede73c-f5b1-4625-8073-fc5401026a40-kube-api-access-8g7tp\") pod \"46ede73c-f5b1-4625-8073-fc5401026a40\" (UID: \"46ede73c-f5b1-4625-8073-fc5401026a40\") " Sep 19 11:04:57.715687 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.715521 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j4mm\" (UniqueName: \"kubernetes.io/projected/b9af77da-8c73-477a-9db8-1f12e966f951-kube-api-access-2j4mm\") pod \"b9af77da-8c73-477a-9db8-1f12e966f951\" (UID: \"b9af77da-8c73-477a-9db8-1f12e966f951\") " Sep 19 11:04:57.715687 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.715552 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9af77da-8c73-477a-9db8-1f12e966f951-plugins-config-volume\") pod \"b9af77da-8c73-477a-9db8-1f12e966f951\" (UID: \"b9af77da-8c73-477a-9db8-1f12e966f951\") " Sep 19 11:04:57.715687 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.715587 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ede73c-f5b1-4625-8073-fc5401026a40-plugins-config-volume\") pod \"46ede73c-f5b1-4625-8073-fc5401026a40\" (UID: \"46ede73c-f5b1-4625-8073-fc5401026a40\") " Sep 19 11:04:57.715996 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.715967 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9af77da-8c73-477a-9db8-1f12e966f951-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9af77da-8c73-477a-9db8-1f12e966f951" (UID: "b9af77da-8c73-477a-9db8-1f12e966f951"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:04:57.716114 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.716081 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ede73c-f5b1-4625-8073-fc5401026a40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "46ede73c-f5b1-4625-8073-fc5401026a40" (UID: "46ede73c-f5b1-4625-8073-fc5401026a40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:04:57.717663 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.717642 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ede73c-f5b1-4625-8073-fc5401026a40-kube-api-access-8g7tp" (OuterVolumeSpecName: "kube-api-access-8g7tp") pod "46ede73c-f5b1-4625-8073-fc5401026a40" (UID: "46ede73c-f5b1-4625-8073-fc5401026a40"). InnerVolumeSpecName "kube-api-access-8g7tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:04:57.717745 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.717666 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9af77da-8c73-477a-9db8-1f12e966f951-kube-api-access-2j4mm" (OuterVolumeSpecName: "kube-api-access-2j4mm") pod "b9af77da-8c73-477a-9db8-1f12e966f951" (UID: "b9af77da-8c73-477a-9db8-1f12e966f951"). InnerVolumeSpecName "kube-api-access-2j4mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:04:57.816587 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.816560 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9af77da-8c73-477a-9db8-1f12e966f951-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:04:57.816587 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.816583 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ede73c-f5b1-4625-8073-fc5401026a40-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:04:57.816587 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.816592 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g7tp\" (UniqueName: \"kubernetes.io/projected/46ede73c-f5b1-4625-8073-fc5401026a40-kube-api-access-8g7tp\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:04:57.816781 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:57.816601 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j4mm\" (UniqueName: \"kubernetes.io/projected/b9af77da-8c73-477a-9db8-1f12e966f951-kube-api-access-2j4mm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:04:58.195495 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:58.195464 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8c80a1c-e9e2-4556-80c1-75fcc6c1913f" path="/var/lib/kubelet/pods/b8c80a1c-e9e2-4556-80c1-75fcc6c1913f/volumes" Sep 19 11:04:58.195802 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:58.195787 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf786a48-3f02-4fe9-9915-082985c38b39" path="/var/lib/kubelet/pods/cf786a48-3f02-4fe9-9915-082985c38b39/volumes" Sep 19 11:04:58.489693 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:58.489603 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" Sep 19 11:04:58.489693 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:58.489645 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28" event={"ID":"b9af77da-8c73-477a-9db8-1f12e966f951","Type":"ContainerDied","Data":"2b194800d946ba0b26ed4222d267692628933ca064cad65264f97e8014270ce7"} Sep 19 11:04:58.490224 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:58.489718 2579 scope.go:117] "RemoveContainer" containerID="a08c561eda8273dbc9a30f074f6b809a76f914ec81987b2e1193745a36ad349c" Sep 19 11:04:58.491903 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:58.491553 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" event={"ID":"46ede73c-f5b1-4625-8073-fc5401026a40","Type":"ContainerDied","Data":"6f43ffc7f4268e2ef444e32245b21b0d3b0157a7449a1d7c6b7e1fca7bc32d60"} Sep 19 11:04:58.491903 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:58.491561 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb" Sep 19 11:04:58.498872 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:58.498857 2579 scope.go:117] "RemoveContainer" containerID="071de16aa68c6ac8ca62d4f93c28e24639683d434290acb361f7d0df07b0313e" Sep 19 11:04:58.510158 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:58.510137 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28"] Sep 19 11:04:58.513960 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:58.513941 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28"] Sep 19 11:04:58.522737 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:58.522719 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb"] Sep 19 11:04:58.526251 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:04:58.526232 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb"] Sep 19 11:05:00.196201 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:00.196169 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46ede73c-f5b1-4625-8073-fc5401026a40" path="/var/lib/kubelet/pods/46ede73c-f5b1-4625-8073-fc5401026a40/volumes" Sep 19 11:05:00.196571 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:00.196523 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9af77da-8c73-477a-9db8-1f12e966f951" path="/var/lib/kubelet/pods/b9af77da-8c73-477a-9db8-1f12e966f951/volumes" Sep 19 11:05:03.962769 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:03.962728 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt"] Sep 19 11:05:03.963272 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:03.963041 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerName="istio-proxy" containerID="cri-o://43e301298b321935a26450b703c1da20cb6053e6576a68d78c1a71bf9217008a" gracePeriod=30 Sep 19 11:05:04.310475 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.310401 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5"] Sep 19 11:05:04.310865 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.310846 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9af77da-8c73-477a-9db8-1f12e966f951" containerName="payload-pre-processing" Sep 19 11:05:04.310953 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.310868 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9af77da-8c73-477a-9db8-1f12e966f951" containerName="payload-pre-processing" Sep 19 11:05:04.310953 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.310884 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ede73c-f5b1-4625-8073-fc5401026a40" containerName="payload-processing" Sep 19 11:05:04.310953 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.310895 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ede73c-f5b1-4625-8073-fc5401026a40" containerName="payload-processing" Sep 19 11:05:04.310953 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.310916 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c80a1c-e9e2-4556-80c1-75fcc6c1913f" containerName="revoke-keys" Sep 19 11:05:04.310953 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.310926 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c80a1c-e9e2-4556-80c1-75fcc6c1913f" containerName="revoke-keys" Sep 19 11:05:04.311162 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.311001 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ede73c-f5b1-4625-8073-fc5401026a40" containerName="payload-processing" Sep 19 11:05:04.311162 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.311011 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9af77da-8c73-477a-9db8-1f12e966f951" containerName="payload-pre-processing" Sep 19 11:05:04.311162 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.311020 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c80a1c-e9e2-4556-80c1-75fcc6c1913f" containerName="revoke-keys" Sep 19 11:05:04.315548 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.315528 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5" Sep 19 11:05:04.318013 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.317990 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 19 11:05:04.318396 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.318319 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mckfb\"" Sep 19 11:05:04.321454 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.321432 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5"] Sep 19 11:05:04.478219 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.478187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f106db3d-497c-4888-b9eb-b8eafab43803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5\" (UID: \"f106db3d-497c-4888-b9eb-b8eafab43803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5" Sep 19 11:05:04.579003 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.578913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f106db3d-497c-4888-b9eb-b8eafab43803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5\" (UID: \"f106db3d-497c-4888-b9eb-b8eafab43803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5" Sep 19 11:05:04.579587 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.579567 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f106db3d-497c-4888-b9eb-b8eafab43803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5\" (UID: \"f106db3d-497c-4888-b9eb-b8eafab43803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5" Sep 19 11:05:04.628361 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.628329 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5" Sep 19 11:05:04.901719 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:04.901688 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5"] Sep 19 11:05:04.903455 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:05:04.903431 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf106db3d_497c_4888_b9eb_b8eafab43803.slice/crio-81dbe2e56d0a6a0f4f0872010f39a88d2a2b4c2c98ad058971ed44837fefdcc5 WatchSource:0}: Error finding container 81dbe2e56d0a6a0f4f0872010f39a88d2a2b4c2c98ad058971ed44837fefdcc5: Status 404 returned error can't find the container with id 81dbe2e56d0a6a0f4f0872010f39a88d2a2b4c2c98ad058971ed44837fefdcc5 Sep 19 11:05:05.534620 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:05.534521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5" event={"ID":"f106db3d-497c-4888-b9eb-b8eafab43803","Type":"ContainerStarted","Data":"ead8516a45b66fad304392845b21879372a8fb3bc260aa174efb43572ad8712c"} Sep 19 11:05:05.534620 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:05.534577 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5" event={"ID":"f106db3d-497c-4888-b9eb-b8eafab43803","Type":"ContainerStarted","Data":"81dbe2e56d0a6a0f4f0872010f39a88d2a2b4c2c98ad058971ed44837fefdcc5"} Sep 19 11:05:05.549005 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:05.548955 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5" podStartSLOduration=1.548937661 podStartE2EDuration="1.548937661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:05:05.548284852 +0000 UTC m=+1837.460789570" watchObservedRunningTime="2026-09-19 11:05:05.548937661 +0000 UTC m=+1837.461442411" Sep 19 11:05:06.543715 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:06.543678 2579 generic.go:356] "Generic (PLEG): container finished" podID="f106db3d-497c-4888-b9eb-b8eafab43803" containerID="ead8516a45b66fad304392845b21879372a8fb3bc260aa174efb43572ad8712c" exitCode=0 Sep 19 11:05:06.544107 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:06.543777 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5" event={"ID":"f106db3d-497c-4888-b9eb-b8eafab43803","Type":"ContainerDied","Data":"ead8516a45b66fad304392845b21879372a8fb3bc260aa174efb43572ad8712c"} Sep 19 11:05:06.589754 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:06.589716 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 19 11:05:06.589937 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:06.589775 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 11:05:07.681450 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:07.681427 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5" Sep 19 11:05:07.712725 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:07.712704 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f106db3d-497c-4888-b9eb-b8eafab43803-maas-api-service-ca\") pod \"f106db3d-497c-4888-b9eb-b8eafab43803\" (UID: \"f106db3d-497c-4888-b9eb-b8eafab43803\") " Sep 19 11:05:07.713018 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:07.712997 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f106db3d-497c-4888-b9eb-b8eafab43803-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f106db3d-497c-4888-b9eb-b8eafab43803" (UID: "f106db3d-497c-4888-b9eb-b8eafab43803"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:05:07.813556 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:07.813500 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f106db3d-497c-4888-b9eb-b8eafab43803-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:08.556590 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:08.556506 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5" Sep 19 11:05:08.556590 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:08.556543 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5" event={"ID":"f106db3d-497c-4888-b9eb-b8eafab43803","Type":"ContainerDied","Data":"81dbe2e56d0a6a0f4f0872010f39a88d2a2b4c2c98ad058971ed44837fefdcc5"} Sep 19 11:05:08.556827 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:08.556592 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81dbe2e56d0a6a0f4f0872010f39a88d2a2b4c2c98ad058971ed44837fefdcc5" Sep 19 11:05:09.221870 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.221845 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 11:05:09.323831 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.323755 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-podinfo\") pod \"2b5096ee-2458-4ac4-9f7c-701756174f18\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " Sep 19 11:05:09.323831 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.323802 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-token\") pod \"2b5096ee-2458-4ac4-9f7c-701756174f18\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " Sep 19 11:05:09.323831 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.323821 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-envoy\") pod \"2b5096ee-2458-4ac4-9f7c-701756174f18\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " Sep 19 11:05:09.324110 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.323860 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pvjk\" (UniqueName: \"kubernetes.io/projected/2b5096ee-2458-4ac4-9f7c-701756174f18-kube-api-access-4pvjk\") pod \"2b5096ee-2458-4ac4-9f7c-701756174f18\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " Sep 19 11:05:09.324110 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.323902 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-credential-socket\") pod \"2b5096ee-2458-4ac4-9f7c-701756174f18\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " Sep 19 11:05:09.324110 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.324090 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-workload-certs\") pod \"2b5096ee-2458-4ac4-9f7c-701756174f18\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " Sep 19 11:05:09.324274 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.324135 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-data\") pod \"2b5096ee-2458-4ac4-9f7c-701756174f18\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " Sep 19 11:05:09.324274 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.324190 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5096ee-2458-4ac4-9f7c-701756174f18-istiod-ca-cert\") pod \"2b5096ee-2458-4ac4-9f7c-701756174f18\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " Sep 19 11:05:09.324274 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.324233 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2b5096ee-2458-4ac4-9f7c-701756174f18" (UID: "2b5096ee-2458-4ac4-9f7c-701756174f18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:05:09.324433 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.324248 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-workload-socket\") pod \"2b5096ee-2458-4ac4-9f7c-701756174f18\" (UID: \"2b5096ee-2458-4ac4-9f7c-701756174f18\") " Sep 19 11:05:09.324433 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.324310 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2b5096ee-2458-4ac4-9f7c-701756174f18" (UID: "2b5096ee-2458-4ac4-9f7c-701756174f18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:05:09.324542 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.324458 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2b5096ee-2458-4ac4-9f7c-701756174f18" (UID: "2b5096ee-2458-4ac4-9f7c-701756174f18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:05:09.324542 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.324489 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-data" (OuterVolumeSpecName: "istio-data") pod "2b5096ee-2458-4ac4-9f7c-701756174f18" (UID: "2b5096ee-2458-4ac4-9f7c-701756174f18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:05:09.324680 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.324656 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5096ee-2458-4ac4-9f7c-701756174f18-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2b5096ee-2458-4ac4-9f7c-701756174f18" (UID: "2b5096ee-2458-4ac4-9f7c-701756174f18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:05:09.324739 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.324664 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:09.324739 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.324710 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:09.324739 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.324726 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:09.324900 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.324741 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:09.326160 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.326133 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2b5096ee-2458-4ac4-9f7c-701756174f18" (UID: "2b5096ee-2458-4ac4-9f7c-701756174f18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:05:09.326160 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.326146 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5096ee-2458-4ac4-9f7c-701756174f18-kube-api-access-4pvjk" (OuterVolumeSpecName: "kube-api-access-4pvjk") pod "2b5096ee-2458-4ac4-9f7c-701756174f18" (UID: "2b5096ee-2458-4ac4-9f7c-701756174f18"). InnerVolumeSpecName "kube-api-access-4pvjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:05:09.326382 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.326368 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-token" (OuterVolumeSpecName: "istio-token") pod "2b5096ee-2458-4ac4-9f7c-701756174f18" (UID: "2b5096ee-2458-4ac4-9f7c-701756174f18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:05:09.326562 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.326535 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2b5096ee-2458-4ac4-9f7c-701756174f18" (UID: "2b5096ee-2458-4ac4-9f7c-701756174f18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 19 11:05:09.425270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.425246 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pvjk\" (UniqueName: \"kubernetes.io/projected/2b5096ee-2458-4ac4-9f7c-701756174f18-kube-api-access-4pvjk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:09.425270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.425265 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5096ee-2458-4ac4-9f7c-701756174f18-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:09.425425 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.425277 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:09.425425 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.425285 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:09.425425 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.425292 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5096ee-2458-4ac4-9f7c-701756174f18-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:09.563789 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.563759 2579 generic.go:356] "Generic (PLEG): container finished" podID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerID="43e301298b321935a26450b703c1da20cb6053e6576a68d78c1a71bf9217008a" exitCode=0 Sep 19 11:05:09.563952 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.563813 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" Sep 19 11:05:09.563952 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.563832 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" event={"ID":"2b5096ee-2458-4ac4-9f7c-701756174f18","Type":"ContainerDied","Data":"43e301298b321935a26450b703c1da20cb6053e6576a68d78c1a71bf9217008a"} Sep 19 11:05:09.563952 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.563873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt" event={"ID":"2b5096ee-2458-4ac4-9f7c-701756174f18","Type":"ContainerDied","Data":"2d0ed5f61cc92f66cac6f5c830e6701a70b22b14f43e418081367c3780d9a79e"} Sep 19 11:05:09.563952 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.563894 2579 scope.go:117] "RemoveContainer" containerID="43e301298b321935a26450b703c1da20cb6053e6576a68d78c1a71bf9217008a" Sep 19 11:05:09.573786 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.573769 2579 scope.go:117] "RemoveContainer" containerID="806fae03de6b58ae179776ea51843ad6b0ab62c54bb3aeb8ee21f52e6b786906" Sep 19 11:05:09.582011 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.581987 2579 scope.go:117] "RemoveContainer" containerID="43e301298b321935a26450b703c1da20cb6053e6576a68d78c1a71bf9217008a" Sep 19 11:05:09.582430 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:05:09.582385 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e301298b321935a26450b703c1da20cb6053e6576a68d78c1a71bf9217008a\": container with ID starting with 43e301298b321935a26450b703c1da20cb6053e6576a68d78c1a71bf9217008a not found: ID does not exist" containerID="43e301298b321935a26450b703c1da20cb6053e6576a68d78c1a71bf9217008a" Sep 19 11:05:09.582627 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.582444 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e301298b321935a26450b703c1da20cb6053e6576a68d78c1a71bf9217008a"} err="failed to get container status \"43e301298b321935a26450b703c1da20cb6053e6576a68d78c1a71bf9217008a\": rpc error: code = NotFound desc = could not find container \"43e301298b321935a26450b703c1da20cb6053e6576a68d78c1a71bf9217008a\": container with ID starting with 43e301298b321935a26450b703c1da20cb6053e6576a68d78c1a71bf9217008a not found: ID does not exist" Sep 19 11:05:09.582627 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.582503 2579 scope.go:117] "RemoveContainer" containerID="806fae03de6b58ae179776ea51843ad6b0ab62c54bb3aeb8ee21f52e6b786906" Sep 19 11:05:09.582978 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:05:09.582938 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806fae03de6b58ae179776ea51843ad6b0ab62c54bb3aeb8ee21f52e6b786906\": container with ID starting with 806fae03de6b58ae179776ea51843ad6b0ab62c54bb3aeb8ee21f52e6b786906 not found: ID does not exist" containerID="806fae03de6b58ae179776ea51843ad6b0ab62c54bb3aeb8ee21f52e6b786906" Sep 19 11:05:09.582978 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.582973 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806fae03de6b58ae179776ea51843ad6b0ab62c54bb3aeb8ee21f52e6b786906"} err="failed to get container status \"806fae03de6b58ae179776ea51843ad6b0ab62c54bb3aeb8ee21f52e6b786906\": rpc error: code = NotFound desc = could not find container \"806fae03de6b58ae179776ea51843ad6b0ab62c54bb3aeb8ee21f52e6b786906\": container with ID starting with 806fae03de6b58ae179776ea51843ad6b0ab62c54bb3aeb8ee21f52e6b786906 not found: ID does not exist" Sep 19 11:05:09.585036 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.585015 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt"] Sep 19 11:05:09.588978 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:09.588957 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt"] Sep 19 11:05:10.197112 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:10.197081 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" path="/var/lib/kubelet/pods/2b5096ee-2458-4ac4-9f7c-701756174f18/volumes" Sep 19 11:05:14.739911 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:14.739879 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn"] Sep 19 11:05:14.742753 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:14.742726 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-99b43e-29830260-5z4qn"] Sep 19 11:05:14.763931 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:14.763907 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh"] Sep 19 11:05:14.764140 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:14.764117 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" podUID="aba8bb4c-a739-480c-9ae5-8d3213c86abc" containerName="payload-processing" containerID="cri-o://8796c3b7e0ed715736e1ae3200fbd28d0a54d2d5381ba627e913fe31fe74f68e" gracePeriod=30 Sep 19 11:05:14.769467 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:14.769438 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb"] Sep 19 11:05:14.769702 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:14.769653 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" podUID="06314f8b-a1f2-44f9-ab95-52cc7919e6fd" containerName="payload-pre-processing" containerID="cri-o://2672ca8a71da71b9d47d2ff7432f9a87d440f15946d9c590ee88d540d464a541" gracePeriod=30 Sep 19 11:05:14.975297 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:14.975248 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5"] Sep 19 11:05:14.978314 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:14.978286 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-99b43e-v6mv5"] Sep 19 11:05:15.045263 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.045230 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" Sep 19 11:05:15.048686 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.048662 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" Sep 19 11:05:15.071791 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.071765 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba8bb4c-a739-480c-9ae5-8d3213c86abc-plugins-config-volume\") pod \"aba8bb4c-a739-480c-9ae5-8d3213c86abc\" (UID: \"aba8bb4c-a739-480c-9ae5-8d3213c86abc\") " Sep 19 11:05:15.071924 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.071818 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06314f8b-a1f2-44f9-ab95-52cc7919e6fd-plugins-config-volume\") pod \"06314f8b-a1f2-44f9-ab95-52cc7919e6fd\" (UID: \"06314f8b-a1f2-44f9-ab95-52cc7919e6fd\") " Sep 19 11:05:15.071924 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.071906 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x96rw\" (UniqueName: \"kubernetes.io/projected/06314f8b-a1f2-44f9-ab95-52cc7919e6fd-kube-api-access-x96rw\") pod \"06314f8b-a1f2-44f9-ab95-52cc7919e6fd\" (UID: \"06314f8b-a1f2-44f9-ab95-52cc7919e6fd\") " Sep 19 11:05:15.072033 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.071942 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9jz4\" (UniqueName: \"kubernetes.io/projected/aba8bb4c-a739-480c-9ae5-8d3213c86abc-kube-api-access-p9jz4\") pod \"aba8bb4c-a739-480c-9ae5-8d3213c86abc\" (UID: \"aba8bb4c-a739-480c-9ae5-8d3213c86abc\") " Sep 19 11:05:15.072176 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.072148 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba8bb4c-a739-480c-9ae5-8d3213c86abc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aba8bb4c-a739-480c-9ae5-8d3213c86abc" (UID: "aba8bb4c-a739-480c-9ae5-8d3213c86abc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:05:15.072263 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.072238 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06314f8b-a1f2-44f9-ab95-52cc7919e6fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06314f8b-a1f2-44f9-ab95-52cc7919e6fd" (UID: "06314f8b-a1f2-44f9-ab95-52cc7919e6fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 11:05:15.072324 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.072267 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba8bb4c-a739-480c-9ae5-8d3213c86abc-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:15.074033 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.074008 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06314f8b-a1f2-44f9-ab95-52cc7919e6fd-kube-api-access-x96rw" (OuterVolumeSpecName: "kube-api-access-x96rw") pod "06314f8b-a1f2-44f9-ab95-52cc7919e6fd" (UID: "06314f8b-a1f2-44f9-ab95-52cc7919e6fd"). InnerVolumeSpecName "kube-api-access-x96rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:05:15.074142 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.074078 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba8bb4c-a739-480c-9ae5-8d3213c86abc-kube-api-access-p9jz4" (OuterVolumeSpecName: "kube-api-access-p9jz4") pod "aba8bb4c-a739-480c-9ae5-8d3213c86abc" (UID: "aba8bb4c-a739-480c-9ae5-8d3213c86abc"). InnerVolumeSpecName "kube-api-access-p9jz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:05:15.173013 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.172982 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x96rw\" (UniqueName: \"kubernetes.io/projected/06314f8b-a1f2-44f9-ab95-52cc7919e6fd-kube-api-access-x96rw\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:15.173013 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.173007 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9jz4\" (UniqueName: \"kubernetes.io/projected/aba8bb4c-a739-480c-9ae5-8d3213c86abc-kube-api-access-p9jz4\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:15.173013 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.173017 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06314f8b-a1f2-44f9-ab95-52cc7919e6fd-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:05:15.600604 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.600569 2579 generic.go:356] "Generic (PLEG): container finished" podID="06314f8b-a1f2-44f9-ab95-52cc7919e6fd" containerID="2672ca8a71da71b9d47d2ff7432f9a87d440f15946d9c590ee88d540d464a541" exitCode=0 Sep 19 11:05:15.600819 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.600627 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" Sep 19 11:05:15.600819 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.600669 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" event={"ID":"06314f8b-a1f2-44f9-ab95-52cc7919e6fd","Type":"ContainerDied","Data":"2672ca8a71da71b9d47d2ff7432f9a87d440f15946d9c590ee88d540d464a541"} Sep 19 11:05:15.600819 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.600741 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb" event={"ID":"06314f8b-a1f2-44f9-ab95-52cc7919e6fd","Type":"ContainerDied","Data":"cc682012ab3f8d489a911069eba0fb9c33e8b4352e0d3c6f3da595fb80dfc759"} Sep 19 11:05:15.600819 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.600775 2579 scope.go:117] "RemoveContainer" containerID="2672ca8a71da71b9d47d2ff7432f9a87d440f15946d9c590ee88d540d464a541" Sep 19 11:05:15.602344 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.602315 2579 generic.go:356] "Generic (PLEG): container finished" podID="aba8bb4c-a739-480c-9ae5-8d3213c86abc" containerID="8796c3b7e0ed715736e1ae3200fbd28d0a54d2d5381ba627e913fe31fe74f68e" exitCode=0 Sep 19 11:05:15.602462 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.602364 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" Sep 19 11:05:15.602462 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.602370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" event={"ID":"aba8bb4c-a739-480c-9ae5-8d3213c86abc","Type":"ContainerDied","Data":"8796c3b7e0ed715736e1ae3200fbd28d0a54d2d5381ba627e913fe31fe74f68e"} Sep 19 11:05:15.602462 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.602409 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh" event={"ID":"aba8bb4c-a739-480c-9ae5-8d3213c86abc","Type":"ContainerDied","Data":"e1bf8380f9a64aeebb3fa3f759ef6cf2a2956d324dde47c755b3a59bbcd2fff9"} Sep 19 11:05:15.610183 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.610165 2579 scope.go:117] "RemoveContainer" containerID="2672ca8a71da71b9d47d2ff7432f9a87d440f15946d9c590ee88d540d464a541" Sep 19 11:05:15.610439 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:05:15.610422 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2672ca8a71da71b9d47d2ff7432f9a87d440f15946d9c590ee88d540d464a541\": container with ID starting with 2672ca8a71da71b9d47d2ff7432f9a87d440f15946d9c590ee88d540d464a541 not found: ID does not exist" containerID="2672ca8a71da71b9d47d2ff7432f9a87d440f15946d9c590ee88d540d464a541" Sep 19 11:05:15.610521 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.610444 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2672ca8a71da71b9d47d2ff7432f9a87d440f15946d9c590ee88d540d464a541"} err="failed to get container status \"2672ca8a71da71b9d47d2ff7432f9a87d440f15946d9c590ee88d540d464a541\": rpc error: code = NotFound desc = could not find container \"2672ca8a71da71b9d47d2ff7432f9a87d440f15946d9c590ee88d540d464a541\": container with ID starting with 2672ca8a71da71b9d47d2ff7432f9a87d440f15946d9c590ee88d540d464a541 not found: ID does not exist" Sep 19 11:05:15.610521 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.610459 2579 scope.go:117] "RemoveContainer" containerID="8796c3b7e0ed715736e1ae3200fbd28d0a54d2d5381ba627e913fe31fe74f68e" Sep 19 11:05:15.618661 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.618646 2579 scope.go:117] "RemoveContainer" containerID="8796c3b7e0ed715736e1ae3200fbd28d0a54d2d5381ba627e913fe31fe74f68e" Sep 19 11:05:15.618881 ip-10-0-143-38 kubenswrapper[2579]: E0919 11:05:15.618864 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8796c3b7e0ed715736e1ae3200fbd28d0a54d2d5381ba627e913fe31fe74f68e\": container with ID starting with 8796c3b7e0ed715736e1ae3200fbd28d0a54d2d5381ba627e913fe31fe74f68e not found: ID does not exist" containerID="8796c3b7e0ed715736e1ae3200fbd28d0a54d2d5381ba627e913fe31fe74f68e" Sep 19 11:05:15.618924 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.618886 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8796c3b7e0ed715736e1ae3200fbd28d0a54d2d5381ba627e913fe31fe74f68e"} err="failed to get container status \"8796c3b7e0ed715736e1ae3200fbd28d0a54d2d5381ba627e913fe31fe74f68e\": rpc error: code = NotFound desc = could not find container \"8796c3b7e0ed715736e1ae3200fbd28d0a54d2d5381ba627e913fe31fe74f68e\": container with ID starting with 8796c3b7e0ed715736e1ae3200fbd28d0a54d2d5381ba627e913fe31fe74f68e not found: ID does not exist" Sep 19 11:05:15.624571 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.624551 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb"] Sep 19 11:05:15.628295 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.628276 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb"] Sep 19 11:05:15.636953 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.636931 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh"] Sep 19 11:05:15.642038 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:15.642016 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh"] Sep 19 11:05:16.196040 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:16.196009 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06314f8b-a1f2-44f9-ab95-52cc7919e6fd" path="/var/lib/kubelet/pods/06314f8b-a1f2-44f9-ab95-52cc7919e6fd/volumes" Sep 19 11:05:16.196454 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:16.196327 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53e12cde-d509-47b2-a6c5-f1e25b9db808" path="/var/lib/kubelet/pods/53e12cde-d509-47b2-a6c5-f1e25b9db808/volumes" Sep 19 11:05:16.196607 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:16.196594 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aba8bb4c-a739-480c-9ae5-8d3213c86abc" path="/var/lib/kubelet/pods/aba8bb4c-a739-480c-9ae5-8d3213c86abc/volumes" Sep 19 11:05:16.196877 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:16.196864 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f106db3d-497c-4888-b9eb-b8eafab43803" path="/var/lib/kubelet/pods/f106db3d-497c-4888-b9eb-b8eafab43803/volumes" Sep 19 11:05:30.611212 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:05:30.611178 2579 scope.go:117] "RemoveContainer" containerID="0190e33bacf489c92ecacaab15a27343853ff2960d4e80164fb4f4933f68078a" Sep 19 11:06:30.657698 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:06:30.657664 2579 scope.go:117] "RemoveContainer" containerID="5bc396061d2b819e2b00641b70b71c9238087f12d680f57dde562647ce2d4e64" Sep 19 11:06:30.666078 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:06:30.666038 2579 scope.go:117] "RemoveContainer" containerID="99262bd826c8842746e4fddfdd3fa825f045473f97474167686824b2f349fa24" Sep 19 11:06:30.673444 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:06:30.673429 2579 scope.go:117] "RemoveContainer" containerID="47a353c0c76a9ff43290618dc627f392c710ff1e92d3f3e6798dc04812bfcfcf" Sep 19 11:06:30.680546 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:06:30.680529 2579 scope.go:117] "RemoveContainer" containerID="26ef5059518242a93d2cb8c72631e9f2801456ec52dd9c0000c60f1365e135c0" Sep 19 11:06:30.687486 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:06:30.687471 2579 scope.go:117] "RemoveContainer" containerID="8c639e7d34862b726b9e6a441419d291680d08b814a09c760724b29a054692f5" Sep 19 11:06:30.694345 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:06:30.694325 2579 scope.go:117] "RemoveContainer" containerID="af05e0993154e1bc6e3001d78c42b035997d50d578f4fffa3f9986abeb702e1d" Sep 19 11:07:55.822410 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:55.822379 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv"] Sep 19 11:07:55.822933 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:55.822561 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" podUID="462e5c82-85a0-4144-94bb-4ed3682f10f2" containerName="manager" containerID="cri-o://dc52d2e760cf2f242f9a36c61e8b25531219fc118aafd1229afc62452006717d" gracePeriod=10 Sep 19 11:07:57.427521 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:57.427493 2579 generic.go:356] "Generic (PLEG): container finished" podID="462e5c82-85a0-4144-94bb-4ed3682f10f2" containerID="dc52d2e760cf2f242f9a36c61e8b25531219fc118aafd1229afc62452006717d" exitCode=0 Sep 19 11:07:57.427848 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:57.427565 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" event={"ID":"462e5c82-85a0-4144-94bb-4ed3682f10f2","Type":"ContainerDied","Data":"dc52d2e760cf2f242f9a36c61e8b25531219fc118aafd1229afc62452006717d"} Sep 19 11:07:57.479116 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:57.479090 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" Sep 19 11:07:57.566288 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:57.566258 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/462e5c82-85a0-4144-94bb-4ed3682f10f2-extensions-socket-volume\") pod \"462e5c82-85a0-4144-94bb-4ed3682f10f2\" (UID: \"462e5c82-85a0-4144-94bb-4ed3682f10f2\") " Sep 19 11:07:57.566437 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:57.566307 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqqq2\" (UniqueName: \"kubernetes.io/projected/462e5c82-85a0-4144-94bb-4ed3682f10f2-kube-api-access-vqqq2\") pod \"462e5c82-85a0-4144-94bb-4ed3682f10f2\" (UID: \"462e5c82-85a0-4144-94bb-4ed3682f10f2\") " Sep 19 11:07:57.566693 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:57.566666 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462e5c82-85a0-4144-94bb-4ed3682f10f2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "462e5c82-85a0-4144-94bb-4ed3682f10f2" (UID: "462e5c82-85a0-4144-94bb-4ed3682f10f2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 19 11:07:57.568375 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:57.568354 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462e5c82-85a0-4144-94bb-4ed3682f10f2-kube-api-access-vqqq2" (OuterVolumeSpecName: "kube-api-access-vqqq2") pod "462e5c82-85a0-4144-94bb-4ed3682f10f2" (UID: "462e5c82-85a0-4144-94bb-4ed3682f10f2"). InnerVolumeSpecName "kube-api-access-vqqq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 11:07:57.667101 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:57.667050 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/462e5c82-85a0-4144-94bb-4ed3682f10f2-extensions-socket-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:07:57.667101 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:57.667097 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqqq2\" (UniqueName: \"kubernetes.io/projected/462e5c82-85a0-4144-94bb-4ed3682f10f2-kube-api-access-vqqq2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\"" Sep 19 11:07:58.438089 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:58.438037 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" event={"ID":"462e5c82-85a0-4144-94bb-4ed3682f10f2","Type":"ContainerDied","Data":"ce694643ce3ce271f16d321dc3c8dc4c4dcd5d5fad3fe1b88cc794413ec8032c"} Sep 19 11:07:58.438474 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:58.438133 2579 scope.go:117] "RemoveContainer" containerID="dc52d2e760cf2f242f9a36c61e8b25531219fc118aafd1229afc62452006717d" Sep 19 11:07:58.438474 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:58.438070 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv" Sep 19 11:07:58.456237 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:58.456184 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv"] Sep 19 11:07:58.459359 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:07:58.459339 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-58svv"] Sep 19 11:08:00.197855 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:00.197823 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="462e5c82-85a0-4144-94bb-4ed3682f10f2" path="/var/lib/kubelet/pods/462e5c82-85a0-4144-94bb-4ed3682f10f2/volumes" Sep 19 11:08:56.510855 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.510761 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm"] Sep 19 11:08:56.511270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511175 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerName="istio-proxy" Sep 19 11:08:56.511270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511191 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerName="istio-proxy" Sep 19 11:08:56.511270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511202 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462e5c82-85a0-4144-94bb-4ed3682f10f2" containerName="manager" Sep 19 11:08:56.511270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511208 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="462e5c82-85a0-4144-94bb-4ed3682f10f2" containerName="manager" Sep 19 11:08:56.511270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511214 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerName="istio-proxy" Sep 19 11:08:56.511270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511220 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerName="istio-proxy" Sep 19 11:08:56.511270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511232 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba8bb4c-a739-480c-9ae5-8d3213c86abc" containerName="payload-processing" Sep 19 11:08:56.511270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511238 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba8bb4c-a739-480c-9ae5-8d3213c86abc" containerName="payload-processing" Sep 19 11:08:56.511270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511245 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06314f8b-a1f2-44f9-ab95-52cc7919e6fd" containerName="payload-pre-processing" Sep 19 11:08:56.511270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511250 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="06314f8b-a1f2-44f9-ab95-52cc7919e6fd" containerName="payload-pre-processing" Sep 19 11:08:56.511270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511259 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f106db3d-497c-4888-b9eb-b8eafab43803" containerName="revoke-keys" Sep 19 11:08:56.511270 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511267 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f106db3d-497c-4888-b9eb-b8eafab43803" containerName="revoke-keys" Sep 19 11:08:56.511648 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511323 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="462e5c82-85a0-4144-94bb-4ed3682f10f2" containerName="manager" Sep 19 11:08:56.511648 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511329 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba8bb4c-a739-480c-9ae5-8d3213c86abc" containerName="payload-processing" Sep 19 11:08:56.511648 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511334 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f106db3d-497c-4888-b9eb-b8eafab43803" containerName="revoke-keys" Sep 19 11:08:56.511648 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511342 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerName="istio-proxy" Sep 19 11:08:56.511648 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511349 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="06314f8b-a1f2-44f9-ab95-52cc7919e6fd" containerName="payload-pre-processing" Sep 19 11:08:56.511648 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.511359 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b5096ee-2458-4ac4-9f7c-701756174f18" containerName="istio-proxy" Sep 19 11:08:56.514303 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.514287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm" Sep 19 11:08:56.516733 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.516712 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 19 11:08:56.516868 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.516791 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 19 11:08:56.516868 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.516796 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-smhjc\"" Sep 19 11:08:56.525406 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.525385 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm"] Sep 19 11:08:56.568397 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.568369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsktj\" (UniqueName: \"kubernetes.io/projected/91632bd9-5467-4f19-8817-95f3ce89b0b2-kube-api-access-nsktj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zqjnm\" (UID: \"91632bd9-5467-4f19-8817-95f3ce89b0b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm" Sep 19 11:08:56.568505 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.568406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91632bd9-5467-4f19-8817-95f3ce89b0b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zqjnm\" (UID: \"91632bd9-5467-4f19-8817-95f3ce89b0b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm" Sep 19 11:08:56.669234 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.669210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsktj\" (UniqueName: \"kubernetes.io/projected/91632bd9-5467-4f19-8817-95f3ce89b0b2-kube-api-access-nsktj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zqjnm\" (UID: \"91632bd9-5467-4f19-8817-95f3ce89b0b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm" Sep 19 11:08:56.669335 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.669243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91632bd9-5467-4f19-8817-95f3ce89b0b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zqjnm\" (UID: \"91632bd9-5467-4f19-8817-95f3ce89b0b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm" Sep 19 11:08:56.669563 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.669547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91632bd9-5467-4f19-8817-95f3ce89b0b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zqjnm\" (UID: \"91632bd9-5467-4f19-8817-95f3ce89b0b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm" Sep 19 11:08:56.678309 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.678282 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsktj\" (UniqueName: \"kubernetes.io/projected/91632bd9-5467-4f19-8817-95f3ce89b0b2-kube-api-access-nsktj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zqjnm\" (UID: \"91632bd9-5467-4f19-8817-95f3ce89b0b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm" Sep 19 11:08:56.825857 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:56.825785 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm" Sep 19 11:08:57.102141 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:57.102114 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm"] Sep 19 11:08:57.107197 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:08:57.107157 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91632bd9_5467_4f19_8817_95f3ce89b0b2.slice/crio-838d124907a79c96553c851015719e1298173fcf980220f4b844cf16bb468abd WatchSource:0}: Error finding container 838d124907a79c96553c851015719e1298173fcf980220f4b844cf16bb468abd: Status 404 returned error can't find the container with id 838d124907a79c96553c851015719e1298173fcf980220f4b844cf16bb468abd Sep 19 11:08:57.728825 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:57.728792 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm" event={"ID":"91632bd9-5467-4f19-8817-95f3ce89b0b2","Type":"ContainerStarted","Data":"b97ed4291d689c6c4a7fc9887d02d7a465e1667bb4ff28dbf0eef2dd071558b6"} Sep 19 11:08:57.729212 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:57.728847 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm" event={"ID":"91632bd9-5467-4f19-8817-95f3ce89b0b2","Type":"ContainerStarted","Data":"838d124907a79c96553c851015719e1298173fcf980220f4b844cf16bb468abd"} Sep 19 11:08:57.729212 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:57.728924 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm" Sep 19 11:08:57.748037 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:08:57.747993 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm" podStartSLOduration=1.747977311 podStartE2EDuration="1.747977311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:08:57.746459443 +0000 UTC m=+2069.658964160" watchObservedRunningTime="2026-09-19 11:08:57.747977311 +0000 UTC m=+2069.660482028" Sep 19 11:09:08.735918 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:09:08.735882 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zqjnm" Sep 19 11:09:30.362316 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:09:30.362287 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 11:09:30.365661 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:09:30.365642 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 11:09:30.758151 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:09:30.758116 2579 scope.go:117] "RemoveContainer" containerID="747295ff7b2dfcb3b6dfc0c2dc0af8c714718b7a99415d3631f0e35b119bee41" Sep 19 11:09:30.773673 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:09:30.773641 2579 scope.go:117] "RemoveContainer" containerID="b12edc58514626f33bfe918b8c9a29d3c387862ae1780a89207f01ecd8f1fcec" Sep 19 11:10:30.795488 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:10:30.795460 2579 scope.go:117] "RemoveContainer" containerID="99959df5f65ced36f97a2ca9a445fffcf351e226b072637ea1cafa0d70579ef8" Sep 19 11:11:16.799533 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:16.799491 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-m7hz5_9dd3d0c6-e1f0-4381-acfa-535576a57096/manager/0.log" Sep 19 11:11:17.140528 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:17.140494 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b87889df-xw7hf_f9aca349-9040-4593-989f-0fdc14b0fe0a/manager/0.log" Sep 19 11:11:17.145631 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:17.145611 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b87889df-xw7hf_f9aca349-9040-4593-989f-0fdc14b0fe0a/copy-manifests/0.log" Sep 19 11:11:17.810176 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:17.810147 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29830260-ldkmf_c85c29cc-2f3c-4ff4-968b-e9d86275eed8/cleanup/0.log" Sep 19 11:11:19.110223 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:19.110196 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-zqjnm_91632bd9-5467-4f19-8817-95f3ce89b0b2/manager/0.log" Sep 19 11:11:19.679804 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:19.679773 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd_da0da6f6-fe34-4d35-90b8-f8bc487946c0/istio-proxy/0.log" Sep 19 11:11:19.802050 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:19.802020 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mjlb7_ade9616b-a22e-4326-8cf8-1fb942ae6084/discovery/0.log" Sep 19 11:11:20.239857 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:20.239823 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-dkm2p_324db52d-7f53-4736-a870-200921c89361/payload-pre-processing/0.log" Sep 19 11:11:20.454378 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:20.454346 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9998785cb-l2lkw_6b92f4b1-f19a-42ac-b87b-835f23d214cd/router/0.log" Sep 19 11:11:21.134565 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:21.134525 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-9fhrj_1de0ea4e-a064-4046-a797-f691acd64344/main/0.log" Sep 19 11:11:21.344075 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:21.344043 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-2nzkx_5407dfe0-508b-464b-8c4f-fab89f076018/main/0.log" Sep 19 11:11:29.246235 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:29.246188 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-m7hz5_9dd3d0c6-e1f0-4381-acfa-535576a57096/manager/0.log" Sep 19 11:11:29.619433 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:29.619359 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b87889df-xw7hf_f9aca349-9040-4593-989f-0fdc14b0fe0a/copy-manifests/0.log" Sep 19 11:11:29.624860 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:29.624833 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b87889df-xw7hf_f9aca349-9040-4593-989f-0fdc14b0fe0a/manager/0.log" Sep 19 11:11:30.276287 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:30.276258 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29830260-ldkmf_c85c29cc-2f3c-4ff4-968b-e9d86275eed8/cleanup/0.log" Sep 19 11:11:30.815205 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:30.815175 2579 scope.go:117] "RemoveContainer" containerID="317ffb8ab3a43c8508e727707c09ae93d3694753a3898dfcb7fbd383f3e3428b" Sep 19 11:11:30.823218 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:30.823200 2579 scope.go:117] "RemoveContainer" containerID="ead8516a45b66fad304392845b21879372a8fb3bc260aa174efb43572ad8712c" Sep 19 11:11:31.589076 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:31.589031 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-zqjnm_91632bd9-5467-4f19-8817-95f3ce89b0b2/manager/0.log" Sep 19 11:11:32.193842 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:32.193813 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd_da0da6f6-fe34-4d35-90b8-f8bc487946c0/istio-proxy/0.log" Sep 19 11:11:32.314049 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:32.314021 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mjlb7_ade9616b-a22e-4326-8cf8-1fb942ae6084/discovery/0.log" Sep 19 11:11:32.771807 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:32.771778 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-dkm2p_324db52d-7f53-4736-a870-200921c89361/payload-pre-processing/0.log" Sep 19 11:11:33.023514 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:33.023424 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9998785cb-l2lkw_6b92f4b1-f19a-42ac-b87b-835f23d214cd/router/0.log" Sep 19 11:11:33.726959 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:33.726931 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-9fhrj_1de0ea4e-a064-4046-a797-f691acd64344/main/0.log" Sep 19 11:11:33.943895 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:33.943867 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-2nzkx_5407dfe0-508b-464b-8c4f-fab89f076018/main/0.log" Sep 19 11:11:55.660070 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:55.659980 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d2djw_f4345b3c-abd0-4cc2-8b21-a87b10324f78/global-pull-secret-syncer/0.log" Sep 19 11:11:55.762238 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:55.762211 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jzzq5_d60a105e-478f-4a4b-b1b9-daff4de47c93/konnectivity-agent/0.log" Sep 19 11:11:55.861138 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:11:55.861104 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-38.ec2.internal_54f93cd73487bad8dda8b78673c5c557/haproxy/0.log" Sep 19 11:12:01.291115 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:01.291003 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-zqjnm_91632bd9-5467-4f19-8817-95f3ce89b0b2/manager/0.log" Sep 19 11:12:02.849645 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:02.849612 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-9rtmm_245f29a4-837b-4197-9eb7-52f324d83901/cluster-monitoring-operator/0.log" Sep 19 11:12:02.872800 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:02.872772 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-wzdmh_72a1da39-f5b2-46e2-a508-9d3b71eed28e/kube-state-metrics/0.log" Sep 19 11:12:02.891843 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:02.891824 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-wzdmh_72a1da39-f5b2-46e2-a508-9d3b71eed28e/kube-rbac-proxy-main/0.log" Sep 19 11:12:02.915573 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:02.915555 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-wzdmh_72a1da39-f5b2-46e2-a508-9d3b71eed28e/kube-rbac-proxy-self/0.log" Sep 19 11:12:03.020041 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:03.019983 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bkpcd_75607ff6-c200-4323-9865-263ec4057fef/node-exporter/0.log" Sep 19 11:12:03.042880 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:03.042860 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bkpcd_75607ff6-c200-4323-9865-263ec4057fef/kube-rbac-proxy/0.log" Sep 19 11:12:03.067176 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:03.067158 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bkpcd_75607ff6-c200-4323-9865-263ec4057fef/init-textfile/0.log" Sep 19 11:12:03.303034 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:03.302963 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0d32aa7-c4a7-423f-836b-b348353e8dea/prometheus/0.log" Sep 19 11:12:03.333999 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:03.333976 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0d32aa7-c4a7-423f-836b-b348353e8dea/config-reloader/0.log" Sep 19 11:12:03.354759 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:03.354741 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0d32aa7-c4a7-423f-836b-b348353e8dea/thanos-sidecar/0.log" Sep 19 11:12:03.375445 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:03.375424 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0d32aa7-c4a7-423f-836b-b348353e8dea/kube-rbac-proxy-web/0.log" Sep 19 11:12:03.395743 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:03.395723 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0d32aa7-c4a7-423f-836b-b348353e8dea/kube-rbac-proxy/0.log" Sep 19 11:12:03.414708 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:03.414687 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0d32aa7-c4a7-423f-836b-b348353e8dea/kube-rbac-proxy-thanos/0.log" Sep 19 11:12:03.434428 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:03.434407 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0d32aa7-c4a7-423f-836b-b348353e8dea/init-config-reloader/0.log" Sep 19 11:12:03.460212 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:03.460186 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-xxr9c_49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1/prometheus-operator/0.log" Sep 19 11:12:03.477394 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:03.477374 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-xxr9c_49e6bdc8-aa3e-4432-8ae1-f35708c6e8d1/kube-rbac-proxy/0.log" Sep 19 11:12:03.503677 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:03.503637 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-f6lqn_d6b50e84-4a6f-469c-bbab-e8524c838662/prometheus-operator-admission-webhook/0.log" Sep 19 11:12:04.947165 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:04.947136 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-7jsd8_e1e97edd-62b0-443a-b1cc-d7b796704644/networking-console-plugin/0.log" Sep 19 11:12:05.592292 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.592262 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/2.log" Sep 19 11:12:05.596074 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.596043 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xnm4h_61826a91-39c7-4d1b-a0f9-c1de1de49c0e/console-operator/3.log" Sep 19 11:12:05.741499 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.741466 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr"] Sep 19 11:12:05.744832 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.744810 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.747125 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.747106 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mc4ck\"/\"default-dockercfg-rpzfz\"" Sep 19 11:12:05.747256 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.747226 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mc4ck\"/\"openshift-service-ca.crt\"" Sep 19 11:12:05.748090 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.748053 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mc4ck\"/\"kube-root-ca.crt\"" Sep 19 11:12:05.752922 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.752900 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr"] Sep 19 11:12:05.809647 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.809623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78b2ae54-5339-42ce-b978-9f676bbd9a6a-podres\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.809784 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.809679 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78b2ae54-5339-42ce-b978-9f676bbd9a6a-lib-modules\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.809784 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.809723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7dbt\" (UniqueName: \"kubernetes.io/projected/78b2ae54-5339-42ce-b978-9f676bbd9a6a-kube-api-access-b7dbt\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.809784 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.809771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78b2ae54-5339-42ce-b978-9f676bbd9a6a-proc\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.809934 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.809802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78b2ae54-5339-42ce-b978-9f676bbd9a6a-sys\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.910536 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.910511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78b2ae54-5339-42ce-b978-9f676bbd9a6a-podres\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.910668 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.910559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78b2ae54-5339-42ce-b978-9f676bbd9a6a-lib-modules\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.910668 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.910591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7dbt\" (UniqueName: \"kubernetes.io/projected/78b2ae54-5339-42ce-b978-9f676bbd9a6a-kube-api-access-b7dbt\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.910668 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.910646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78b2ae54-5339-42ce-b978-9f676bbd9a6a-proc\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.910835 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.910679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78b2ae54-5339-42ce-b978-9f676bbd9a6a-sys\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.910835 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.910685 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78b2ae54-5339-42ce-b978-9f676bbd9a6a-podres\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.910835 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.910741 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78b2ae54-5339-42ce-b978-9f676bbd9a6a-lib-modules\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.910835 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.910779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78b2ae54-5339-42ce-b978-9f676bbd9a6a-sys\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.910835 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.910785 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78b2ae54-5339-42ce-b978-9f676bbd9a6a-proc\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:05.917865 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:05.917843 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7dbt\" (UniqueName: \"kubernetes.io/projected/78b2ae54-5339-42ce-b978-9f676bbd9a6a-kube-api-access-b7dbt\") pod \"perf-node-gather-daemonset-bdhbr\" (UID: \"78b2ae54-5339-42ce-b978-9f676bbd9a6a\") " pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:06.055225 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:06.055200 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:06.327168 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:06.327140 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr"] Sep 19 11:12:06.329465 ip-10-0-143-38 kubenswrapper[2579]: W0919 11:12:06.329423 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod78b2ae54_5339_42ce_b978_9f676bbd9a6a.slice/crio-543aaac8d844d4c4ab33e2afbff612fa4a8049ad4b552efbdec92fd3b82c6c20 WatchSource:0}: Error finding container 543aaac8d844d4c4ab33e2afbff612fa4a8049ad4b552efbdec92fd3b82c6c20: Status 404 returned error can't find the container with id 543aaac8d844d4c4ab33e2afbff612fa4a8049ad4b552efbdec92fd3b82c6c20 Sep 19 11:12:06.650818 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:06.650782 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" event={"ID":"78b2ae54-5339-42ce-b978-9f676bbd9a6a","Type":"ContainerStarted","Data":"6f6649cd5d45ba89476cc78f9ee3a581dcd0ab8cfbdb4c2b4315a8c689c3db95"} Sep 19 11:12:06.651025 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:06.650836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" event={"ID":"78b2ae54-5339-42ce-b978-9f676bbd9a6a","Type":"ContainerStarted","Data":"543aaac8d844d4c4ab33e2afbff612fa4a8049ad4b552efbdec92fd3b82c6c20"} Sep 19 11:12:06.651025 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:06.650891 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:06.665396 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:06.665348 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" podStartSLOduration=1.665332435 podStartE2EDuration="1.665332435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:12:06.664026611 +0000 UTC m=+2258.576531340" watchObservedRunningTime="2026-09-19 11:12:06.665332435 +0000 UTC m=+2258.577837183" Sep 19 11:12:06.839561 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:06.839532 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4md95_697fd709-5743-4f20-aa9b-8cc6285edeaa/volume-data-source-validator/0.log" Sep 19 11:12:07.862706 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:07.862674 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gvn74_6a046e7f-51bb-42f3-be5a-e3f15a80afce/dns/0.log" Sep 19 11:12:07.880016 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:07.879993 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gvn74_6a046e7f-51bb-42f3-be5a-e3f15a80afce/kube-rbac-proxy/0.log" Sep 19 11:12:07.979228 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:07.979199 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c9c22_9e5b260f-bb0a-4096-9a9f-f4a534e77eea/dns-node-resolver/0.log" Sep 19 11:12:08.538093 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:08.538049 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k2ddk_747ac66b-6228-471f-9409-77c6246911dd/node-ca/0.log" Sep 19 11:12:09.450171 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:09.450144 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd_da0da6f6-fe34-4d35-90b8-f8bc487946c0/istio-proxy/0.log" Sep 19 11:12:09.776953 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:09.776878 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mjlb7_ade9616b-a22e-4326-8cf8-1fb942ae6084/discovery/0.log" Sep 19 11:12:09.874114 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:09.874087 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-dkm2p_324db52d-7f53-4736-a870-200921c89361/payload-pre-processing/0.log" Sep 19 11:12:09.922715 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:09.922686 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9998785cb-l2lkw_6b92f4b1-f19a-42ac-b87b-835f23d214cd/router/0.log" Sep 19 11:12:10.759100 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:10.759072 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4cgqn_c425dda8-86ce-4a68-947b-e32ded00bfba/serve-healthcheck-canary/0.log" Sep 19 11:12:11.220263 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:11.220234 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-t5cft_efe9b96b-1362-4fc4-8701-36225c008ec9/insights-operator/0.log" Sep 19 11:12:11.220486 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:11.220469 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-t5cft_efe9b96b-1362-4fc4-8701-36225c008ec9/insights-operator/1.log" Sep 19 11:12:11.350138 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:11.350115 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pq7d5_2f036b1a-5177-4773-b2d1-d2537cf6d5e1/kube-rbac-proxy/0.log" Sep 19 11:12:11.368034 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:11.368016 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pq7d5_2f036b1a-5177-4773-b2d1-d2537cf6d5e1/exporter/0.log" Sep 19 11:12:11.386710 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:11.386683 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pq7d5_2f036b1a-5177-4773-b2d1-d2537cf6d5e1/extractor/0.log" Sep 19 11:12:12.664645 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:12.664613 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mc4ck/perf-node-gather-daemonset-bdhbr" Sep 19 11:12:14.052991 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:14.052963 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-m7hz5_9dd3d0c6-e1f0-4381-acfa-535576a57096/manager/0.log" Sep 19 11:12:14.210745 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:14.210718 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b87889df-xw7hf_f9aca349-9040-4593-989f-0fdc14b0fe0a/manager/0.log" Sep 19 11:12:14.228948 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:14.228919 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b87889df-xw7hf_f9aca349-9040-4593-989f-0fdc14b0fe0a/copy-manifests/0.log" Sep 19 11:12:15.700997 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:15.700965 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c7d4469d9-wpkp5_fbcd0ddf-e8d9-4a89-b535-6ce173163f54/manager/0.log" Sep 19 11:12:16.158970 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:16.158936 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd_da0da6f6-fe34-4d35-90b8-f8bc487946c0/istio-proxy/0.log" Sep 19 11:12:16.490187 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:16.490118 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mjlb7_ade9616b-a22e-4326-8cf8-1fb942ae6084/discovery/0.log" Sep 19 11:12:16.551584 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:16.551559 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-dkm2p_324db52d-7f53-4736-a870-200921c89361/payload-pre-processing/0.log" Sep 19 11:12:16.581842 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:16.581814 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9998785cb-l2lkw_6b92f4b1-f19a-42ac-b87b-835f23d214cd/router/0.log" Sep 19 11:12:18.189149 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:18.189119 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd_da0da6f6-fe34-4d35-90b8-f8bc487946c0/istio-proxy/0.log" Sep 19 11:12:18.549436 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:18.549411 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mjlb7_ade9616b-a22e-4326-8cf8-1fb942ae6084/discovery/0.log" Sep 19 11:12:18.622388 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:18.622367 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-dkm2p_324db52d-7f53-4736-a870-200921c89361/payload-pre-processing/0.log" Sep 19 11:12:18.662532 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:18.662506 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9998785cb-l2lkw_6b92f4b1-f19a-42ac-b87b-835f23d214cd/router/0.log" Sep 19 11:12:20.965800 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:20.965775 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-2jc62_bdae529f-3e44-4813-906d-fafa5aa85f27/kube-storage-version-migrator-operator/1.log" Sep 19 11:12:20.966541 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:20.966525 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-2jc62_bdae529f-3e44-4813-906d-fafa5aa85f27/kube-storage-version-migrator-operator/0.log" Sep 19 11:12:21.896648 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:21.896622 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jrfz_5bcf3b15-117f-4aaf-a94a-08997ee640c8/kube-multus-additional-cni-plugins/0.log" Sep 19 11:12:21.914494 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:21.914469 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jrfz_5bcf3b15-117f-4aaf-a94a-08997ee640c8/egress-router-binary-copy/0.log" Sep 19 11:12:21.933426 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:21.933408 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jrfz_5bcf3b15-117f-4aaf-a94a-08997ee640c8/cni-plugins/0.log" Sep 19 11:12:21.950935 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:21.950911 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jrfz_5bcf3b15-117f-4aaf-a94a-08997ee640c8/bond-cni-plugin/0.log" Sep 19 11:12:21.967832 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:21.967814 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jrfz_5bcf3b15-117f-4aaf-a94a-08997ee640c8/routeoverride-cni/0.log" Sep 19 11:12:21.984851 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:21.984832 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jrfz_5bcf3b15-117f-4aaf-a94a-08997ee640c8/whereabouts-cni-bincopy/0.log" Sep 19 11:12:22.003013 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:22.002990 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jrfz_5bcf3b15-117f-4aaf-a94a-08997ee640c8/whereabouts-cni/0.log" Sep 19 11:12:22.311889 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:22.311814 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dpm4k_712bf5a4-5979-4e67-bab1-92d6befbf2e8/kube-multus/0.log" Sep 19 11:12:22.422756 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:22.422735 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9xvrh_1728a47c-9260-477e-8b40-42c3cfdee6db/network-metrics-daemon/0.log" Sep 19 11:12:22.437665 ip-10-0-143-38 kubenswrapper[2579]: I0919 11:12:22.437643 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9xvrh_1728a47c-9260-477e-8b40-42c3cfdee6db/kube-rbac-proxy/0.log"