Sep 10 23:34:38.027212 ip-10-0-134-90 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 23:34:38.027226 ip-10-0-134-90 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 23:34:38.027236 ip-10-0-134-90 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 23:34:38.027437 ip-10-0-134-90 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 23:34:48.276169 ip-10-0-134-90 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 10 23:34:48.276186 ip-10-0-134-90 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d55ded0909e14abc8e868c6d99e32970 -- Sep 10 23:36:32.571349 ip-10-0-134-90 systemd[1]: Starting Kubernetes Kubelet... Sep 10 23:36:33.008387 ip-10-0-134-90 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 23:36:33.008387 ip-10-0-134-90 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 23:36:33.008387 ip-10-0-134-90 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 23:36:33.008387 ip-10-0-134-90 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 23:36:33.008387 ip-10-0-134-90 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 23:36:33.010518 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.010432 2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 23:36:33.015406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015391 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:33.015406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015405 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015409 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015412 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015416 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015419 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015423 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015426 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015428 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015431 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015434 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015437 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015440 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015442 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015445 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015447 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015450 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015453 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015455 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015459 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015462 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:33.015471 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015464 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015467 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015470 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015474 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015476 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015479 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015482 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015485 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015489 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015493 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015496 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015499 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015501 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015504 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015506 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015509 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015512 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015515 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015518 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:33.015952 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015520 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015523 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015525 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015528 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015530 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015533 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015535 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015538 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015540 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015543 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015545 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015549 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015551 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015554 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015557 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015560 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015563 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015566 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015569 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:33.016431 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015571 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015574 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015578 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015581 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015584 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015587 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015590 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015593 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015595 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015598 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015600 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015604 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015607 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015610 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015612 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015615 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015617 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015620 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015623 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:33.016885 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015627 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015631 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015634 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015637 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015640 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015643 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015645 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015648 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015650 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015654 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015657 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015659 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015662 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015665 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015667 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015670 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015672 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015675 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015678 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015680 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:33.017406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.015683 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016382 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016389 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016392 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016394 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016397 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016401 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016404 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016406 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016409 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016412 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016415 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016417 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016420 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016423 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016426 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016428 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016431 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016433 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016436 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:33.017884 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016439 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016442 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016445 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016447 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016450 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016453 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016455 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016458 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016460 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016463 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016466 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016468 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016471 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016473 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016476 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016478 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016481 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016483 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016486 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016489 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:33.018406 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016492 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016494 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016497 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016501 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016504 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016506 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016509 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016511 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016514 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016516 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016519 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016522 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016524 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016527 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016530 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016533 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016535 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016538 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016541 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016543 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:33.018894 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016546 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016550 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016552 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016555 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016557 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016560 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016563 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016566 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016568 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016571 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016573 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016576 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016584 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016587 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016591 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016595 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016598 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016602 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016606 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:33.019398 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016609 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016611 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016616 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016620 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016623 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016626 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016629 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016632 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016635 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016638 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016641 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016644 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016647 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016650 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016653 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016656 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016658 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016661 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016663 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016666 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:33.019861 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.016670 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017100 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017106 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017109 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017112 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017118 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017121 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017124 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017127 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017129 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017132 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017135 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017137 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017140 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017142 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017145 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017147 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017150 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017154 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017156 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:33.020365 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017159 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017161 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017164 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017167 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017170 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017172 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017175 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017178 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017180 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017183 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017185 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017188 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017191 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017194 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017197 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017200 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017204 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017206 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017209 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017212 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:33.020851 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017214 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017217 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017219 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017222 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017225 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017227 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017230 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017232 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017235 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017237 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017241 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017243 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017246 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017249 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017251 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017254 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017256 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017259 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017261 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017264 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:33.021452 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017266 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017269 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017272 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017275 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017278 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017280 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017283 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017286 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017290 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017293 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017295 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017298 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017301 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017304 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017307 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017311 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017314 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017316 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017319 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:33.021982 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017322 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017325 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017329 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017332 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017335 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017338 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017340 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017343 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017345 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017348 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017352 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017354 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017357 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017361 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017364 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017367 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017369 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017372 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017375 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:33.022465 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017377 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.017380 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017450 2583 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017484 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017491 2583 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017495 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017500 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017504 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017508 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017512 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017515 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017518 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017521 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017525 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017529 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017533 2583 flags.go:64] FLAG: --cgroup-root="" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017536 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017539 2583 flags.go:64] FLAG: --client-ca-file="" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017542 2583 flags.go:64] FLAG: --cloud-provider="external" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017545 2583 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017549 2583 flags.go:64] FLAG: --cluster-domain="" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017552 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017555 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017558 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 23:36:33.022913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017561 2583 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017565 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017568 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017571 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017574 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017577 2583 flags.go:64] FLAG: --contention-profiling="false" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017580 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017583 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017587 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017590 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017594 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017597 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017600 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017603 2583 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017606 2583 flags.go:64] FLAG: --enable-server="true" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017609 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017616 2583 flags.go:64] FLAG: --event-burst="100" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017619 2583 flags.go:64] FLAG: --event-qps="50" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017622 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017625 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017628 2583 flags.go:64] FLAG: --eviction-hard="" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017632 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017635 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017639 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017642 2583 flags.go:64] FLAG: --eviction-soft="" Sep 10 23:36:33.023514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017645 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017648 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017652 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017655 2583 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017658 2583 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017660 2583 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017663 2583 flags.go:64] FLAG: --feature-gates="" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017667 2583 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017670 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017673 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017676 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017679 2583 flags.go:64] FLAG: --healthz-port="10248" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017682 2583 flags.go:64] FLAG: --help="false" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017685 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017688 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017691 2583 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017694 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017697 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017700 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017703 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017706 2583 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017709 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017712 2583 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 23:36:33.024146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017715 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017719 2583 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017722 2583 flags.go:64] FLAG: --kube-reserved="" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017725 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017729 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017732 2583 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017735 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017740 2583 flags.go:64] FLAG: --lock-file="" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017743 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017746 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017749 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017755 2583 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017758 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017760 2583 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017763 2583 flags.go:64] FLAG: --logging-format="text" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017766 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017769 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017772 2583 flags.go:64] FLAG: --manifest-url="" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017775 2583 flags.go:64] FLAG: --manifest-url-header="" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017779 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017782 2583 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017786 2583 flags.go:64] FLAG: --max-pods="110" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017789 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017792 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017795 2583 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 23:36:33.024745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017797 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017800 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017803 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017806 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017813 2583 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017816 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017819 2583 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017822 2583 flags.go:64] FLAG: --pod-cidr="" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017825 2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017831 2583 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017834 2583 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017837 2583 flags.go:64] FLAG: --pods-per-core="0" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017841 2583 flags.go:64] FLAG: --port="10250" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017845 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017849 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f0c6f6a4b4dbfc97" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017853 2583 flags.go:64] FLAG: --qos-reserved="" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017856 2583 flags.go:64] FLAG: --read-only-port="10255" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017859 2583 flags.go:64] FLAG: --register-node="true" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017862 2583 flags.go:64] FLAG: --register-with-taints="" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017865 2583 flags.go:64] FLAG: --registry-burst="10" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017868 2583 flags.go:64] FLAG: --registry-qps="5" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017871 2583 flags.go:64] FLAG: --reserved-cpus="" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017874 2583 flags.go:64] FLAG: --reserved-memory="" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017877 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017880 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 23:36:33.025406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017883 2583 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017886 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017889 2583 flags.go:64] FLAG: --runonce="false" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017892 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017895 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017898 2583 flags.go:64] FLAG: --seccomp-default="false" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017901 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017904 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017907 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017910 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017913 2583 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017916 2583 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017919 2583 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017922 2583 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017925 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017928 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017931 2583 flags.go:64] FLAG: --system-cgroups="" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017934 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017939 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017943 2583 flags.go:64] FLAG: --tls-cert-file="" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017946 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017950 2583 flags.go:64] FLAG: --tls-min-version="" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017953 2583 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017956 2583 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017959 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 23:36:33.026012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017962 2583 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017965 2583 flags.go:64] FLAG: --v="2" Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017969 2583 flags.go:64] FLAG: --version="false" Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017973 2583 flags.go:64] FLAG: --vmodule="" Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017977 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.017980 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018110 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018114 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018119 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018123 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018126 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018130 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018132 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018135 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018138 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018140 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018143 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018145 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018148 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018151 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018153 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:33.026629 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018156 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018159 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018161 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018164 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018167 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018172 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018175 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018178 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018182 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018185 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018188 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018191 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018194 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018197 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018199 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018202 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018205 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018208 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018210 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:33.027170 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018213 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018216 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018218 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018221 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018223 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018226 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018229 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018231 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018234 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018236 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018240 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018244 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018246 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018249 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018251 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018254 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018256 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018260 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018264 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018266 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:33.027631 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018269 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018272 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018275 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018278 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018280 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018283 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018286 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018288 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018291 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018294 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018296 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018299 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018301 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018304 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018307 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018309 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018312 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018315 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018317 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018320 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:33.028140 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018322 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018325 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018327 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018330 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018332 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018335 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018337 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018340 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018342 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018345 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018350 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018352 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018355 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018358 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018360 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018363 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018365 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018368 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018370 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:33.028632 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018373 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018376 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018378 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018381 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018383 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.018386 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.019304 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.027383 2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.027399 2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.028328 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.028339 2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028403 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028408 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028411 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:33.029159 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028415 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028418 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028420 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028423 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028426 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028429 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028431 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028434 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028436 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028447 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028450 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028453 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028455 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028458 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028461 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028465 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028468 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028472 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028475 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:33.029521 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028477 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028480 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028483 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028487 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028489 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028492 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028495 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028499 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028502 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028505 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028508 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028511 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028514 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028518 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028521 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028524 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028527 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028529 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028532 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028535 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:33.029984 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028538 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028540 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028543 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028551 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028554 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028556 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028559 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028562 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028565 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028567 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028570 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028572 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028575 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028577 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028580 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028583 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028585 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028588 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028590 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028593 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:33.030487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028596 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028599 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028602 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028604 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028608 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028611 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028614 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028616 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028619 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028621 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028624 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028626 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028629 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028632 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028634 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028637 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028645 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028648 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028650 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:33.030977 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028653 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028656 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028658 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028660 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028663 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028666 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028668 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028671 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028673 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028676 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028679 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028681 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028684 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028687 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028690 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028692 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028695 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:33.031513 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028697 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.028703 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028869 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028874 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028877 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028880 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028883 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028886 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028889 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028892 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028895 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028898 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028901 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028909 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028912 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:33.031916 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028915 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028917 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028920 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028922 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028925 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028928 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028930 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028933 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028937 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028940 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028943 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028946 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028950 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028953 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028955 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028958 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028961 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028964 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028966 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:33.032301 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028969 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028972 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028975 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028977 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028980 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028983 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028985 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028988 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028990 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028992 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028996 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.028998 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029006 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029009 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029011 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029014 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029017 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029019 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029022 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:33.032764 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029024 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029028 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029032 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029035 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029037 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029040 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029044 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029047 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029049 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029052 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029061 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029064 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029079 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029083 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029085 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029088 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029091 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029094 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029096 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029099 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:33.033241 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029102 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029104 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029107 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029110 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029112 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029114 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029123 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029126 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029129 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029131 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029134 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029136 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029139 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029142 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029144 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029147 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029149 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029152 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029155 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029160 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:33.033729 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029164 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:33.034221 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029167 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:33.034221 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029170 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:33.034221 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029172 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:33.034221 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029175 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:33.034221 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029178 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:33.034221 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029181 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:33.034221 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:33.029183 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:33.034221 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.029193 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 23:36:33.034221 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.029295 2583 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 23:36:33.034221 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.031859 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 23:36:33.034221 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.032829 2583 server.go:1019] "Starting client certificate rotation" Sep 10 23:36:33.034221 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.032938 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 23:36:33.034221 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.032967 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 23:36:33.035744 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.035729 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 23:36:33.041116 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.041091 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 23:36:33.049802 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.049786 2583 log.go:25] "Validated CRI v1 runtime API" Sep 10 23:36:33.054331 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.054312 2583 log.go:25] "Validated CRI v1 image API" Sep 10 23:36:33.055127 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.055100 2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 23:36:33.058476 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.058449 2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8efe23ed-7766-4475-96ff-d5ae96bd25fb:/dev/nvme0n1p3 c64669d8-abff-44b1-99de-5ffcd2e88764:/dev/nvme0n1p4] Sep 10 23:36:33.058542 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.058475 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 23:36:33.063529 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.063415 2583 manager.go:216] Machine: {Timestamp:2026-09-10 23:36:33.062011571 +0000 UTC m=+0.364489574 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098380 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dd52d10cfc751e9addb97c4ddc2f3 SystemUUID:ec2dd52d-10cf-c751-e9ad-db97c4ddc2f3 BootID:d55ded09-09e1-4abc-8e86-8c6d99e32970 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:23:83:c3:35:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:23:83:c3:35:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:3e:76:51:e8:28 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 23:36:33.063529 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.063522 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 23:36:33.063632 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.063597 2583 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 23:36:33.065625 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.065611 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 23:36:33.067159 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.067135 2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 23:36:33.067291 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.067163 2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-90.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 23:36:33.068005 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.067994 2583 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 23:36:33.068043 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.068009 2583 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 23:36:33.068043 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.068017 2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 23:36:33.068043 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.068039 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 23:36:33.069361 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.069351 2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 23:36:33.074097 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.074085 2583 state_mem.go:36] "Initialized new in-memory state store" Sep 10 23:36:33.074218 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.074208 2583 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 23:36:33.074247 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.074240 2583 kubelet.go:486] "Attempting to sync node with API server" Sep 10 23:36:33.074278 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.074255 2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 23:36:33.074308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.074278 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 23:36:33.074308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.074287 2583 kubelet.go:398] "Adding apiserver pod source" Sep 10 23:36:33.074308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.074304 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 23:36:33.074756 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.074743 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 23:36:33.074813 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.074768 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 23:36:33.079274 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.079259 2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 23:36:33.084315 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.084290 2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 23:36:33.084393 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.084337 2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 23:36:33.086062 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.086039 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-90.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 23:36:33.086147 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.086063 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 23:36:33.086317 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.086304 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 23:36:33.086377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.086321 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 23:36:33.086377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.086328 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 23:36:33.086377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.086334 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 23:36:33.086377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.086342 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 23:36:33.086377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.086348 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 23:36:33.086377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.086360 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 23:36:33.086377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.086366 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 23:36:33.086377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.086372 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 23:36:33.086377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.086377 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 23:36:33.086607 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.086386 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 23:36:33.086607 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.086391 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 23:36:33.087188 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.087179 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 23:36:33.087188 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.087189 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 23:36:33.090841 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.090829 2583 server.go:1268] "Started kubelet" Sep 10 23:36:33.090934 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.090892 2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 23:36:33.091031 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.090973 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 23:36:33.091110 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.091044 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 23:36:33.091719 ip-10-0-134-90 systemd[1]: Started Kubernetes Kubelet. Sep 10 23:36:33.092489 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.092471 2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 23:36:33.093136 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.093121 2583 server.go:310] "Adding debug handlers to kubelet server" Sep 10 23:36:33.095679 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.095663 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-90.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 23:36:33.096868 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.095827 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-90.ec2.internal.18d419ddf6ba57bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-90.ec2.internal,UID:ip-10-0-134-90.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-90.ec2.internal,},FirstTimestamp:2026-09-10 23:36:33.090803647 +0000 UTC m=+0.393281649,LastTimestamp:2026-09-10 23:36:33.090803647 +0000 UTC m=+0.393281649,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-90.ec2.internal,}" Sep 10 23:36:33.098130 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.097978 2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 23:36:33.099152 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.099135 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tm5pg" Sep 10 23:36:33.099949 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.099932 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 23:36:33.100671 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.100657 2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 23:36:33.101175 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.100801 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-90.ec2.internal\" not found" Sep 10 23:36:33.101411 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.101392 2583 factory.go:55] Registering systemd factory Sep 10 23:36:33.101411 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.101412 2583 factory.go:223] Registration of the systemd container factory successfully Sep 10 23:36:33.101543 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.101497 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 23:36:33.101543 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.101499 2583 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 23:36:33.101543 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.101521 2583 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 23:36:33.101659 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.101646 2583 factory.go:153] Registering CRI-O factory Sep 10 23:36:33.101690 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.101664 2583 factory.go:223] Registration of the crio container factory successfully Sep 10 23:36:33.101721 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.101710 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 23:36:33.101721 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.101652 2583 reconstruct.go:97] "Volume reconstruction finished" Sep 10 23:36:33.101779 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.101724 2583 reconciler.go:29] "Reconciler: start to sync state" Sep 10 23:36:33.101779 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.101728 2583 factory.go:103] Registering Raw factory Sep 10 23:36:33.101779 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.101742 2583 manager.go:1196] Started watching for new ooms in manager Sep 10 23:36:33.102308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.102292 2583 manager.go:349] Starting recovery of all containers Sep 10 23:36:33.104249 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.104233 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tm5pg" Sep 10 23:36:33.109142 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.109105 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 23:36:33.117270 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.117246 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:33.118033 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.118017 2583 manager.go:354] Recovery completed Sep 10 23:36:33.119841 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.119817 2583 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-90.ec2.internal\" not found" node="ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.122723 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.122712 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:33.124330 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.124314 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:33.124386 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.124342 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:33.124386 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.124352 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:33.124875 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.124861 2583 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 23:36:33.124960 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.124875 2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 23:36:33.124960 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.124864 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:33.124960 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.124892 2583 state_mem.go:36] "Initialized new in-memory state store" Sep 10 23:36:33.125581 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.125565 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:33.125651 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.125592 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:33.125651 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.125602 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:33.127163 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.127151 2583 policy_none.go:49] "None policy: Start" Sep 10 23:36:33.127219 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.127168 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 23:36:33.127219 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.127181 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 23:36:33.129283 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.129269 2583 policy_none.go:47] "Start" Sep 10 23:36:33.152661 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.152647 2583 manager.go:337] "Starting Device Plugin manager" Sep 10 23:36:33.152742 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.152717 2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 23:36:33.152742 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.152730 2583 server.go:87] "Starting device plugin registration server" Sep 10 23:36:33.152971 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.152959 2583 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 23:36:33.153131 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.152972 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 23:36:33.153131 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.153089 2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 23:36:33.153374 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.153160 2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 23:36:33.153374 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.153169 2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 23:36:33.153649 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.153626 2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 23:36:33.153730 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.153678 2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-90.ec2.internal\" not found" Sep 10 23:36:33.171065 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.171040 2583 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2619 into the system container \"/system.slice\": " Sep 10 23:36:33.254077 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.254047 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:33.254960 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.254946 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:33.255035 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.254972 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:33.255035 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.254981 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:33.255035 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.255003 2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.263210 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.263124 2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.263210 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.263145 2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-90.ec2.internal\": node \"ip-10-0-134-90.ec2.internal\" not found" Sep 10 23:36:33.275422 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.275403 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-90.ec2.internal\" not found" Sep 10 23:36:33.301047 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.300909 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 23:36:33.306322 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.301089 2583 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 23:36:33.306322 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.301118 2583 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 23:36:33.306322 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.301154 2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 23:36:33.311275 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.311258 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:33.376395 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.376373 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-90.ec2.internal\" not found" Sep 10 23:36:33.401545 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.401520 2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-90.ec2.internal"] Sep 10 23:36:33.401626 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.401598 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:33.402581 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.402567 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:33.402652 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.402594 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:33.402652 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.402606 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:33.404958 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.404945 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:33.405131 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.405118 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.405171 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.405158 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:33.407080 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.407051 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:33.407140 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.407096 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:33.407140 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.407107 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:33.407205 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.407051 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:33.407205 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.407170 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:33.407205 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.407181 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:33.409484 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.409471 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.409537 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.409503 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:33.411370 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.411354 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:33.411449 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.411381 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:33.411449 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.411396 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:33.424599 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.424581 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-90.ec2.internal\" not found" node="ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.429975 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.429960 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-90.ec2.internal\" not found" node="ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.476715 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.476698 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-90.ec2.internal\" not found" Sep 10 23:36:33.577034 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.576960 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-90.ec2.internal\" not found" Sep 10 23:36:33.603302 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.603275 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3779dff9bc8ed2f1165f8546a4a88973-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal\" (UID: \"3779dff9bc8ed2f1165f8546a4a88973\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.603393 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.603311 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3779dff9bc8ed2f1165f8546a4a88973-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal\" (UID: \"3779dff9bc8ed2f1165f8546a4a88973\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.603393 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.603339 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8aff0765baa894ad2af911b31f796a12-config\") pod \"kube-apiserver-proxy-ip-10-0-134-90.ec2.internal\" (UID: \"8aff0765baa894ad2af911b31f796a12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.677430 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.677400 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-90.ec2.internal\" not found" Sep 10 23:36:33.703737 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.703711 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3779dff9bc8ed2f1165f8546a4a88973-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal\" (UID: \"3779dff9bc8ed2f1165f8546a4a88973\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.703795 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.703743 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8aff0765baa894ad2af911b31f796a12-config\") pod \"kube-apiserver-proxy-ip-10-0-134-90.ec2.internal\" (UID: \"8aff0765baa894ad2af911b31f796a12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.703795 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.703757 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3779dff9bc8ed2f1165f8546a4a88973-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal\" (UID: \"3779dff9bc8ed2f1165f8546a4a88973\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.703795 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.703783 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3779dff9bc8ed2f1165f8546a4a88973-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal\" (UID: \"3779dff9bc8ed2f1165f8546a4a88973\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.703912 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.703798 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3779dff9bc8ed2f1165f8546a4a88973-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal\" (UID: \"3779dff9bc8ed2f1165f8546a4a88973\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.703912 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.703820 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8aff0765baa894ad2af911b31f796a12-config\") pod \"kube-apiserver-proxy-ip-10-0-134-90.ec2.internal\" (UID: \"8aff0765baa894ad2af911b31f796a12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.728848 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.728828 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.734257 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:33.734112 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-90.ec2.internal" Sep 10 23:36:33.777771 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.777742 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-90.ec2.internal\" not found" Sep 10 23:36:33.883899 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.883819 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-90.ec2.internal\" not found" Sep 10 23:36:33.984375 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:33.984346 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-90.ec2.internal\" not found" Sep 10 23:36:34.033387 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:34.033362 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 23:36:34.033776 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:34.033518 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 23:36:34.033776 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:34.033540 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 23:36:34.084705 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:34.084667 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-90.ec2.internal\" not found" Sep 10 23:36:34.111358 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:34.111274 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 23:31:33 +0000 UTC" deadline="2028-02-14 17:50:25.395103956 +0000 UTC" Sep 10 23:36:34.111358 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:34.111356 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12522h13m51.283754755s" Sep 10 23:36:34.137575 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:34.137506 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:34.201247 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:34.201219 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal" Sep 10 23:36:34.215360 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:34.215341 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 23:36:34.217584 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:34.217563 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-90.ec2.internal" Sep 10 23:36:34.224519 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:34.224504 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 23:36:34.278631 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:34.278605 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:34.434701 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:34.434679 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 23:36:34.440100 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:34.440061 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3779dff9bc8ed2f1165f8546a4a88973.slice/crio-d793b1c54c100c79894fe55f640c337485505c58684388c7e16e724da90078e8 WatchSource:0}: Error finding container d793b1c54c100c79894fe55f640c337485505c58684388c7e16e724da90078e8: Status 404 returned error can't find the container with id d793b1c54c100c79894fe55f640c337485505c58684388c7e16e724da90078e8 Sep 10 23:36:35.077544 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.077509 2583 apiserver.go:52] "Watching apiserver" Sep 10 23:36:35.088591 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.088560 2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 23:36:35.089180 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.089141 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7","openshift-dns/node-resolver-rvpbw","openshift-image-registry/node-ca-zlqzj","openshift-multus/multus-additional-cni-plugins-pvxvx","openshift-multus/multus-wnl97","openshift-multus/network-metrics-daemon-7hv8v","openshift-network-diagnostics/network-check-target-hxfnv","openshift-network-operator/iptables-alerter-mwtvd","kube-system/global-pull-secret-syncer-rt86f","kube-system/konnectivity-agent-gdmff","kube-system/kube-apiserver-proxy-ip-10-0-134-90.ec2.internal","openshift-cluster-node-tuning-operator/tuned-kvg4b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-rc9rz"] Sep 10 23:36:35.091959 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.091940 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:35.092059 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.092020 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:36:35.094362 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.094334 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.097007 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.096986 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mwtvd" Sep 10 23:36:35.098010 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.097990 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 23:36:35.098207 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.098190 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 23:36:35.098588 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.098568 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ssrcc\"" Sep 10 23:36:35.098934 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.098920 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.100283 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.100265 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 23:36:35.101320 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.101300 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 23:36:35.102277 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.101728 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 23:36:35.102277 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.101755 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 23:36:35.102277 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.101805 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.102277 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.101984 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-szw92\"" Sep 10 23:36:35.104605 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.104279 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:35.104605 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.104354 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:36:35.105547 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.105523 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 23:36:35.106645 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.106624 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 23:36:35.107579 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.107554 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.108632 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.108613 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t8hpb\"" Sep 10 23:36:35.108794 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.108776 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 23:36:35.108922 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.108906 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 23:36:35.109037 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.109022 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 23:36:35.109105 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.108910 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 23:36:35.109105 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.109095 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 23:36:35.109478 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.109460 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vfpbn\"" Sep 10 23:36:35.109543 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.109501 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 23:36:35.110028 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.109992 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zlqzj" Sep 10 23:36:35.110028 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.109931 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 23:36:35.110412 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.110222 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4vds2\"" Sep 10 23:36:35.112679 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112429 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-sysctl-d\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.112679 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112467 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31ca19af-6670-4b07-b1b3-d6737e89a1a4-iptables-alerter-script\") pod \"iptables-alerter-mwtvd\" (UID: \"31ca19af-6670-4b07-b1b3-d6737e89a1a4\") " pod="openshift-network-operator/iptables-alerter-mwtvd" Sep 10 23:36:35.112679 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112493 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs\") pod \"network-metrics-daemon-7hv8v\" (UID: \"e030b330-e967-4317-bcd2-4664364ee362\") " pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:35.112679 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112517 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-modprobe-d\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.112679 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112543 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-kubernetes\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.112679 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112566 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-sys\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.112679 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112589 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-tuned\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.112679 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112612 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k76nw\" (UniqueName: \"kubernetes.io/projected/05dad598-ac28-407d-b6c8-fa493d21c90f-kube-api-access-k76nw\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.112679 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112652 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-device-dir\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.112679 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112681 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-systemd\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112707 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05dad598-ac28-407d-b6c8-fa493d21c90f-tmp\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112730 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crg7h\" (UniqueName: \"kubernetes.io/projected/31ca19af-6670-4b07-b1b3-d6737e89a1a4-kube-api-access-crg7h\") pod \"iptables-alerter-mwtvd\" (UID: \"31ca19af-6670-4b07-b1b3-d6737e89a1a4\") " pod="openshift-network-operator/iptables-alerter-mwtvd" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112763 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-socket-dir\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112788 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-etc-selinux\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112810 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0170eaf4-0b83-4c76-be27-600a18f3226e-system-cni-dir\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112843 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0170eaf4-0b83-4c76-be27-600a18f3226e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112868 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk4zc\" (UniqueName: \"kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc\") pod \"network-check-target-hxfnv\" (UID: \"bf403608-4d7b-4585-88b9-7b1590654d3e\") " pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-sysconfig\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112915 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-lib-modules\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112939 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31ca19af-6670-4b07-b1b3-d6737e89a1a4-host-slash\") pod \"iptables-alerter-mwtvd\" (UID: \"31ca19af-6670-4b07-b1b3-d6737e89a1a4\") " pod="openshift-network-operator/iptables-alerter-mwtvd" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.112980 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0170eaf4-0b83-4c76-be27-600a18f3226e-cnibin\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.113006 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0170eaf4-0b83-4c76-be27-600a18f3226e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.113029 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-run\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.113053 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-host\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.113079 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.113313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.113104 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-registration-dir\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.114117 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.113127 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-var-lib-kubelet\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.114117 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.113168 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-sysctl-conf\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.114117 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.113257 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-sys-fs\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.114117 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.113295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0170eaf4-0b83-4c76-be27-600a18f3226e-cni-binary-copy\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.114117 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.113317 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0170eaf4-0b83-4c76-be27-600a18f3226e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.114117 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.113334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fklsg\" (UniqueName: \"kubernetes.io/projected/e030b330-e967-4317-bcd2-4664364ee362-kube-api-access-fklsg\") pod \"network-metrics-daemon-7hv8v\" (UID: \"e030b330-e967-4317-bcd2-4664364ee362\") " pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:35.114117 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.113367 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shrgv\" (UniqueName: \"kubernetes.io/projected/0170eaf4-0b83-4c76-be27-600a18f3226e-kube-api-access-shrgv\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.114117 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.113395 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5v5b\" (UniqueName: \"kubernetes.io/projected/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-kube-api-access-t5v5b\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.114562 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.114215 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:35.114562 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.114275 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:36:35.116933 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.116910 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gdmff" Sep 10 23:36:35.120320 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.119095 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rvpbw" Sep 10 23:36:35.120320 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.119190 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cmxbz\"" Sep 10 23:36:35.120320 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.119489 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 23:36:35.120320 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.119812 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 23:36:35.120320 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.119916 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7445n\"" Sep 10 23:36:35.120320 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.120071 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 23:36:35.120320 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.120099 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 23:36:35.120320 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.120167 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 23:36:35.122311 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.121607 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x2bxs\"" Sep 10 23:36:35.122311 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.121721 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.122534 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.122484 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 23:36:35.123356 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.123141 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 23:36:35.124777 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.124597 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 23:36:35.124777 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.124653 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 23:36:35.124777 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.124665 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 23:36:35.125301 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.125281 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 23:36:35.126164 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.125678 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 23:36:35.126164 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.125681 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 23:36:35.126164 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.125968 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wx56f\"" Sep 10 23:36:35.130457 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.130437 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 23:36:35.152530 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.152506 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p2bts" Sep 10 23:36:35.161749 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.161727 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p2bts" Sep 10 23:36:35.203349 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.203322 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 23:36:35.214311 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214281 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qms8x\" (UniqueName: \"kubernetes.io/projected/1c0b07bf-513c-4c0f-aed5-bcfd96136254-kube-api-access-qms8x\") pod \"node-resolver-rvpbw\" (UID: \"1c0b07bf-513c-4c0f-aed5-bcfd96136254\") " pod="openshift-dns/node-resolver-rvpbw" Sep 10 23:36:35.214429 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214320 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-kubelet\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.214429 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214350 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-hostroot\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.214429 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214373 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-systemd\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.214429 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214391 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05dad598-ac28-407d-b6c8-fa493d21c90f-tmp\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.214429 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214406 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-socket-dir\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.214429 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214422 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0170eaf4-0b83-4c76-be27-600a18f3226e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214446 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-multus-cni-dir\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214470 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-node-log\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214493 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9-konnectivity-ca\") pod \"konnectivity-agent-gdmff\" (UID: \"79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9\") " pod="kube-system/konnectivity-agent-gdmff" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214503 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-systemd\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214517 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0170eaf4-0b83-4c76-be27-600a18f3226e-cnibin\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214543 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0170eaf4-0b83-4c76-be27-600a18f3226e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214568 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0b07bf-513c-4c0f-aed5-bcfd96136254-tmp-dir\") pod \"node-resolver-rvpbw\" (UID: \"1c0b07bf-513c-4c0f-aed5-bcfd96136254\") " pod="openshift-dns/node-resolver-rvpbw" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214568 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0170eaf4-0b83-4c76-be27-600a18f3226e-cnibin\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214577 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-socket-dir\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214591 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-host\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214638 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9-agent-certs\") pod \"konnectivity-agent-gdmff\" (UID: \"79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9\") " pod="kube-system/konnectivity-agent-gdmff" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214646 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-host\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214675 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3750ae5a-25be-4d56-89ae-67210d6cb357-host\") pod \"node-ca-zlqzj\" (UID: \"3750ae5a-25be-4d56-89ae-67210d6cb357\") " pod="openshift-image-registry/node-ca-zlqzj" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214701 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0331a4a5-abd1-419c-9116-60b56f90673b-ovnkube-script-lib\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214722 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-var-lib-kubelet\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214737 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0170eaf4-0b83-4c76-be27-600a18f3226e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.214748 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214745 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-sys-fs\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214776 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-var-lib-kubelet\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214784 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-sys-fs\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0170eaf4-0b83-4c76-be27-600a18f3226e-cni-binary-copy\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214800 2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214827 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-multus-socket-dir-parent\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214887 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-cni-bin\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214916 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shrgv\" (UniqueName: \"kubernetes.io/projected/0170eaf4-0b83-4c76-be27-600a18f3226e-kube-api-access-shrgv\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214943 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b40d527-4121-4aeb-8abd-c4c7c5f53688-kubelet-config\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214959 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-etc-openvswitch\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214976 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5v5b\" (UniqueName: \"kubernetes.io/projected/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-kube-api-access-t5v5b\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.214998 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-etc-kubernetes\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215024 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-sysctl-d\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215048 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs\") pod \"network-metrics-daemon-7hv8v\" (UID: \"e030b330-e967-4317-bcd2-4664364ee362\") " pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215066 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0170eaf4-0b83-4c76-be27-600a18f3226e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215070 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3750ae5a-25be-4d56-89ae-67210d6cb357-serviceca\") pod \"node-ca-zlqzj\" (UID: \"3750ae5a-25be-4d56-89ae-67210d6cb357\") " pod="openshift-image-registry/node-ca-zlqzj" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215146 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b40d527-4121-4aeb-8abd-c4c7c5f53688-dbus\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:35.215590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215163 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0331a4a5-abd1-419c-9116-60b56f90673b-env-overrides\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215194 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-modprobe-d\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215228 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-kubernetes\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215252 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k76nw\" (UniqueName: \"kubernetes.io/projected/05dad598-ac28-407d-b6c8-fa493d21c90f-kube-api-access-k76nw\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215275 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-device-dir\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215287 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-sysctl-d\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215309 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-run-multus-certs\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215335 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm2b9\" (UniqueName: \"kubernetes.io/projected/0331a4a5-abd1-419c-9116-60b56f90673b-kube-api-access-mm2b9\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215348 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-device-dir\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215362 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-var-lib-cni-multus\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215371 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-modprobe-d\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215379 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-kubernetes\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.215284 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215396 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-var-lib-kubelet\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215433 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crg7h\" (UniqueName: \"kubernetes.io/projected/31ca19af-6670-4b07-b1b3-d6737e89a1a4-kube-api-access-crg7h\") pod \"iptables-alerter-mwtvd\" (UID: \"31ca19af-6670-4b07-b1b3-d6737e89a1a4\") " pod="openshift-network-operator/iptables-alerter-mwtvd" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.215527 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs podName:e030b330-e967-4317-bcd2-4664364ee362 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:35.715499196 +0000 UTC m=+3.017977209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs") pod "network-metrics-daemon-7hv8v" (UID: "e030b330-e967-4317-bcd2-4664364ee362") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215551 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-etc-selinux\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.216505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215585 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0170eaf4-0b83-4c76-be27-600a18f3226e-system-cni-dir\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215612 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4zc\" (UniqueName: \"kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc\") pod \"network-check-target-hxfnv\" (UID: \"bf403608-4d7b-4585-88b9-7b1590654d3e\") " pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215641 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c0b07bf-513c-4c0f-aed5-bcfd96136254-hosts-file\") pod \"node-resolver-rvpbw\" (UID: \"1c0b07bf-513c-4c0f-aed5-bcfd96136254\") " pod="openshift-dns/node-resolver-rvpbw" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215667 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-run-netns\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215693 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-multus-daemon-config\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215720 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-sysconfig\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215749 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-etc-selinux\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215756 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-lib-modules\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215804 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31ca19af-6670-4b07-b1b3-d6737e89a1a4-host-slash\") pod \"iptables-alerter-mwtvd\" (UID: \"31ca19af-6670-4b07-b1b3-d6737e89a1a4\") " pod="openshift-network-operator/iptables-alerter-mwtvd" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215843 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-cnibin\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215871 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-var-lib-cni-bin\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215888 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-lib-modules\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215896 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-slash\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215929 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-sysconfig\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215940 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0331a4a5-abd1-419c-9116-60b56f90673b-ovnkube-config\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215969 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0170eaf4-0b83-4c76-be27-600a18f3226e-system-cni-dir\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215971 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-run\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.217416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.215996 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31ca19af-6670-4b07-b1b3-d6737e89a1a4-host-slash\") pod \"iptables-alerter-mwtvd\" (UID: \"31ca19af-6670-4b07-b1b3-d6737e89a1a4\") " pod="openshift-network-operator/iptables-alerter-mwtvd" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216042 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216070 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-registration-dir\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216101 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216166 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-run\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216166 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-cni-binary-copy\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216180 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-registration-dir\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216214 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216249 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-sysctl-conf\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216291 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0170eaf4-0b83-4c76-be27-600a18f3226e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216326 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fklsg\" (UniqueName: \"kubernetes.io/projected/e030b330-e967-4317-bcd2-4664364ee362-kube-api-access-fklsg\") pod \"network-metrics-daemon-7hv8v\" (UID: \"e030b330-e967-4317-bcd2-4664364ee362\") " pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216363 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-run-netns\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216378 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-sysctl-conf\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216393 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-multus-conf-dir\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216423 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-var-lib-openvswitch\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216445 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0170eaf4-0b83-4c76-be27-600a18f3226e-cni-binary-copy\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216464 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-run-ovn\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.218438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216505 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-log-socket\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216545 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-run-ovn-kubernetes\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216579 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-run-systemd\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216611 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-run-openvswitch\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216641 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31ca19af-6670-4b07-b1b3-d6737e89a1a4-iptables-alerter-script\") pod \"iptables-alerter-mwtvd\" (UID: \"31ca19af-6670-4b07-b1b3-d6737e89a1a4\") " pod="openshift-network-operator/iptables-alerter-mwtvd" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216667 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-system-cni-dir\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216694 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-run-k8s-cni-cncf-io\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216705 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0170eaf4-0b83-4c76-be27-600a18f3226e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216724 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216763 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-systemd-units\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216805 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-cni-netd\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0331a4a5-abd1-419c-9116-60b56f90673b-ovn-node-metrics-cert\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216875 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j54qp\" (UniqueName: \"kubernetes.io/projected/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-kube-api-access-j54qp\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216902 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-sys\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216950 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-tuned\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.216951 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05dad598-ac28-407d-b6c8-fa493d21c90f-sys\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.217059 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dh8h\" (UniqueName: \"kubernetes.io/projected/3750ae5a-25be-4d56-89ae-67210d6cb357-kube-api-access-8dh8h\") pod \"node-ca-zlqzj\" (UID: \"3750ae5a-25be-4d56-89ae-67210d6cb357\") " pod="openshift-image-registry/node-ca-zlqzj" Sep 10 23:36:35.219165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.217212 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31ca19af-6670-4b07-b1b3-d6737e89a1a4-iptables-alerter-script\") pod \"iptables-alerter-mwtvd\" (UID: \"31ca19af-6670-4b07-b1b3-d6737e89a1a4\") " pod="openshift-network-operator/iptables-alerter-mwtvd" Sep 10 23:36:35.221238 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.221102 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05dad598-ac28-407d-b6c8-fa493d21c90f-tmp\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.221305 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.221275 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05dad598-ac28-407d-b6c8-fa493d21c90f-etc-tuned\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.228699 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.228441 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:35.228699 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.228467 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:35.228699 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.228481 2583 projected.go:196] Error preparing data for projected volume kube-api-access-fk4zc for pod openshift-network-diagnostics/network-check-target-hxfnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:35.228699 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.228545 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc podName:bf403608-4d7b-4585-88b9-7b1590654d3e nodeName:}" failed. No retries permitted until 2026-09-10 23:36:35.72852693 +0000 UTC m=+3.031004950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fk4zc" (UniqueName: "kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc") pod "network-check-target-hxfnv" (UID: "bf403608-4d7b-4585-88b9-7b1590654d3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:35.230246 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.230068 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5v5b\" (UniqueName: \"kubernetes.io/projected/2f901d14-3c13-4ea6-8218-3653e4ef6ac7-kube-api-access-t5v5b\") pod \"aws-ebs-csi-driver-node-sxpv7\" (UID: \"2f901d14-3c13-4ea6-8218-3653e4ef6ac7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.230980 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.230955 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shrgv\" (UniqueName: \"kubernetes.io/projected/0170eaf4-0b83-4c76-be27-600a18f3226e-kube-api-access-shrgv\") pod \"multus-additional-cni-plugins-pvxvx\" (UID: \"0170eaf4-0b83-4c76-be27-600a18f3226e\") " pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.232317 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.232294 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k76nw\" (UniqueName: \"kubernetes.io/projected/05dad598-ac28-407d-b6c8-fa493d21c90f-kube-api-access-k76nw\") pod \"tuned-kvg4b\" (UID: \"05dad598-ac28-407d-b6c8-fa493d21c90f\") " pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.232594 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.232570 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crg7h\" (UniqueName: \"kubernetes.io/projected/31ca19af-6670-4b07-b1b3-d6737e89a1a4-kube-api-access-crg7h\") pod \"iptables-alerter-mwtvd\" (UID: \"31ca19af-6670-4b07-b1b3-d6737e89a1a4\") " pod="openshift-network-operator/iptables-alerter-mwtvd" Sep 10 23:36:35.233621 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.233598 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fklsg\" (UniqueName: \"kubernetes.io/projected/e030b330-e967-4317-bcd2-4664364ee362-kube-api-access-fklsg\") pod \"network-metrics-daemon-7hv8v\" (UID: \"e030b330-e967-4317-bcd2-4664364ee362\") " pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:35.307025 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.306963 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-90.ec2.internal" event={"ID":"8aff0765baa894ad2af911b31f796a12","Type":"ContainerStarted","Data":"813daea5f575ad07081a695356f0829c4c94add420c828cce131bc6d1ada3f85"} Sep 10 23:36:35.308640 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.308610 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal" event={"ID":"3779dff9bc8ed2f1165f8546a4a88973","Type":"ContainerStarted","Data":"d793b1c54c100c79894fe55f640c337485505c58684388c7e16e724da90078e8"} Sep 10 23:36:35.317942 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.317914 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-var-lib-cni-multus\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318095 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.317950 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-var-lib-kubelet\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318095 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.317992 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c0b07bf-513c-4c0f-aed5-bcfd96136254-hosts-file\") pod \"node-resolver-rvpbw\" (UID: \"1c0b07bf-513c-4c0f-aed5-bcfd96136254\") " pod="openshift-dns/node-resolver-rvpbw" Sep 10 23:36:35.318095 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318047 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-var-lib-cni-multus\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318095 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318062 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c0b07bf-513c-4c0f-aed5-bcfd96136254-hosts-file\") pod \"node-resolver-rvpbw\" (UID: \"1c0b07bf-513c-4c0f-aed5-bcfd96136254\") " pod="openshift-dns/node-resolver-rvpbw" Sep 10 23:36:35.318265 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318073 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-run-netns\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318265 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318118 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-run-netns\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318265 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318129 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-var-lib-kubelet\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318265 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318133 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-multus-daemon-config\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318265 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318173 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-cnibin\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318265 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318190 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-var-lib-cni-bin\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318265 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318215 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-slash\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318265 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318242 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0331a4a5-abd1-419c-9116-60b56f90673b-ovnkube-config\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318265 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318254 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-cnibin\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318540 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318270 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-cni-binary-copy\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318540 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318297 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318540 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318326 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-run-netns\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318540 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318352 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-multus-conf-dir\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318540 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318351 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-slash\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318540 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318376 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-var-lib-openvswitch\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318540 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318398 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-var-lib-cni-bin\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318540 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318403 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-run-ovn\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318540 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318456 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-log-socket\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318540 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318481 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-run-ovn-kubernetes\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318540 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318529 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-run-ovn-kubernetes\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318540 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318539 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-multus-daemon-config\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318892 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318589 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-var-lib-openvswitch\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318892 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318620 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-run-netns\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318892 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318649 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-multus-conf-dir\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.318892 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318676 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-log-socket\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318892 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318701 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-run-ovn\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318892 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318449 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318892 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318729 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-run-systemd\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318892 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318744 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-run-openvswitch\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.318892 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-system-cni-dir\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.319418 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.318776 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-run-k8s-cni-cncf-io\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.319418 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319056 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:35.319418 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-systemd-units\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.319418 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319162 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-cni-netd\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.319418 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319213 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0331a4a5-abd1-419c-9116-60b56f90673b-ovn-node-metrics-cert\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.319418 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319239 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j54qp\" (UniqueName: \"kubernetes.io/projected/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-kube-api-access-j54qp\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.319418 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319263 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dh8h\" (UniqueName: \"kubernetes.io/projected/3750ae5a-25be-4d56-89ae-67210d6cb357-kube-api-access-8dh8h\") pod \"node-ca-zlqzj\" (UID: \"3750ae5a-25be-4d56-89ae-67210d6cb357\") " pod="openshift-image-registry/node-ca-zlqzj" Sep 10 23:36:35.319418 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319295 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qms8x\" (UniqueName: \"kubernetes.io/projected/1c0b07bf-513c-4c0f-aed5-bcfd96136254-kube-api-access-qms8x\") pod \"node-resolver-rvpbw\" (UID: \"1c0b07bf-513c-4c0f-aed5-bcfd96136254\") " pod="openshift-dns/node-resolver-rvpbw" Sep 10 23:36:35.319418 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319321 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-kubelet\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.319418 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319381 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-hostroot\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.319930 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319448 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-run-systemd\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.319930 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319498 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-run-openvswitch\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.319930 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319805 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0331a4a5-abd1-419c-9116-60b56f90673b-ovnkube-config\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.319930 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319883 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-system-cni-dir\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.320138 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.319936 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-run-k8s-cni-cncf-io\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.320138 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.320052 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:35.320138 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.320121 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret podName:9b40d527-4121-4aeb-8abd-c4c7c5f53688 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:35.820101711 +0000 UTC m=+3.122579720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret") pod "global-pull-secret-syncer-rt86f" (UID: "9b40d527-4121-4aeb-8abd-c4c7c5f53688") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:35.320315 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.320193 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-systemd-units\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.320315 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.320253 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-cni-netd\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.320840 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.320799 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-cni-binary-copy\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.321648 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.321626 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-kubelet\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.321895 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.321875 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-hostroot\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.321950 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.321930 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-multus-cni-dir\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.321990 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.321960 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-node-log\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.322031 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.321996 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9-konnectivity-ca\") pod \"konnectivity-agent-gdmff\" (UID: \"79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9\") " pod="kube-system/konnectivity-agent-gdmff" Sep 10 23:36:35.322073 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322032 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0b07bf-513c-4c0f-aed5-bcfd96136254-tmp-dir\") pod \"node-resolver-rvpbw\" (UID: \"1c0b07bf-513c-4c0f-aed5-bcfd96136254\") " pod="openshift-dns/node-resolver-rvpbw" Sep 10 23:36:35.322109 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322067 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9-agent-certs\") pod \"konnectivity-agent-gdmff\" (UID: \"79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9\") " pod="kube-system/konnectivity-agent-gdmff" Sep 10 23:36:35.322140 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3750ae5a-25be-4d56-89ae-67210d6cb357-host\") pod \"node-ca-zlqzj\" (UID: \"3750ae5a-25be-4d56-89ae-67210d6cb357\") " pod="openshift-image-registry/node-ca-zlqzj" Sep 10 23:36:35.322196 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322149 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0331a4a5-abd1-419c-9116-60b56f90673b-ovnkube-script-lib\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.322196 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322185 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-multus-socket-dir-parent\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.322261 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322218 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-cni-bin\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.322296 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322255 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b40d527-4121-4aeb-8abd-c4c7c5f53688-kubelet-config\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:35.322296 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322285 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-etc-openvswitch\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.322360 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322322 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-etc-kubernetes\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.322391 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322376 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3750ae5a-25be-4d56-89ae-67210d6cb357-serviceca\") pod \"node-ca-zlqzj\" (UID: \"3750ae5a-25be-4d56-89ae-67210d6cb357\") " pod="openshift-image-registry/node-ca-zlqzj" Sep 10 23:36:35.322422 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322403 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b40d527-4121-4aeb-8abd-c4c7c5f53688-dbus\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:35.322459 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322436 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0331a4a5-abd1-419c-9116-60b56f90673b-env-overrides\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.322492 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322465 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-run-multus-certs\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.322492 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322488 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm2b9\" (UniqueName: \"kubernetes.io/projected/0331a4a5-abd1-419c-9116-60b56f90673b-kube-api-access-mm2b9\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.322934 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322916 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-multus-cni-dir\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.323006 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.322967 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-node-log\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.323358 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.323336 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0331a4a5-abd1-419c-9116-60b56f90673b-ovn-node-metrics-cert\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.323512 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.323494 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b40d527-4121-4aeb-8abd-c4c7c5f53688-kubelet-config\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:35.323575 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.323515 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9-konnectivity-ca\") pod \"konnectivity-agent-gdmff\" (UID: \"79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9\") " pod="kube-system/konnectivity-agent-gdmff" Sep 10 23:36:35.323634 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.323577 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-etc-openvswitch\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.323692 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.323655 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-etc-kubernetes\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.323818 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.323798 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0b07bf-513c-4c0f-aed5-bcfd96136254-tmp-dir\") pod \"node-resolver-rvpbw\" (UID: \"1c0b07bf-513c-4c0f-aed5-bcfd96136254\") " pod="openshift-dns/node-resolver-rvpbw" Sep 10 23:36:35.324077 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.324057 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3750ae5a-25be-4d56-89ae-67210d6cb357-serviceca\") pod \"node-ca-zlqzj\" (UID: \"3750ae5a-25be-4d56-89ae-67210d6cb357\") " pod="openshift-image-registry/node-ca-zlqzj" Sep 10 23:36:35.324189 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.324171 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b40d527-4121-4aeb-8abd-c4c7c5f53688-dbus\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:35.324629 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.324607 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0331a4a5-abd1-419c-9116-60b56f90673b-env-overrides\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.324715 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.324692 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-host-run-multus-certs\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.325304 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.325260 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0331a4a5-abd1-419c-9116-60b56f90673b-ovnkube-script-lib\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.325405 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.325330 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3750ae5a-25be-4d56-89ae-67210d6cb357-host\") pod \"node-ca-zlqzj\" (UID: \"3750ae5a-25be-4d56-89ae-67210d6cb357\") " pod="openshift-image-registry/node-ca-zlqzj" Sep 10 23:36:35.325405 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.325400 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-multus-socket-dir-parent\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.325522 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.325461 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0331a4a5-abd1-419c-9116-60b56f90673b-host-cni-bin\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.327039 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.327006 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9-agent-certs\") pod \"konnectivity-agent-gdmff\" (UID: \"79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9\") " pod="kube-system/konnectivity-agent-gdmff" Sep 10 23:36:35.334364 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.334299 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dh8h\" (UniqueName: \"kubernetes.io/projected/3750ae5a-25be-4d56-89ae-67210d6cb357-kube-api-access-8dh8h\") pod \"node-ca-zlqzj\" (UID: \"3750ae5a-25be-4d56-89ae-67210d6cb357\") " pod="openshift-image-registry/node-ca-zlqzj" Sep 10 23:36:35.334768 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.334745 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j54qp\" (UniqueName: \"kubernetes.io/projected/22a3fa41-e1e7-481c-a52a-ffb83f4804cc-kube-api-access-j54qp\") pod \"multus-wnl97\" (UID: \"22a3fa41-e1e7-481c-a52a-ffb83f4804cc\") " pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.335420 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.335398 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qms8x\" (UniqueName: \"kubernetes.io/projected/1c0b07bf-513c-4c0f-aed5-bcfd96136254-kube-api-access-qms8x\") pod \"node-resolver-rvpbw\" (UID: \"1c0b07bf-513c-4c0f-aed5-bcfd96136254\") " pod="openshift-dns/node-resolver-rvpbw" Sep 10 23:36:35.335741 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.335721 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm2b9\" (UniqueName: \"kubernetes.io/projected/0331a4a5-abd1-419c-9116-60b56f90673b-kube-api-access-mm2b9\") pod \"ovnkube-node-rc9rz\" (UID: \"0331a4a5-abd1-419c-9116-60b56f90673b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.379615 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.379589 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:35.407928 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.407894 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" Sep 10 23:36:35.423397 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.423375 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mwtvd" Sep 10 23:36:35.435334 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.435312 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" Sep 10 23:36:35.445215 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.445195 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pvxvx" Sep 10 23:36:35.452175 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.452150 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wnl97" Sep 10 23:36:35.460060 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.460032 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zlqzj" Sep 10 23:36:35.467923 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.467897 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gdmff" Sep 10 23:36:35.476797 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.476772 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rvpbw" Sep 10 23:36:35.483758 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.483727 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:35.635509 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.635426 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:35.726298 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.726269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs\") pod \"network-metrics-daemon-7hv8v\" (UID: \"e030b330-e967-4317-bcd2-4664364ee362\") " pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:35.726453 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.726407 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:35.726500 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.726477 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs podName:e030b330-e967-4317-bcd2-4664364ee362 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:36.726455485 +0000 UTC m=+4.028933481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs") pod "network-metrics-daemon-7hv8v" (UID: "e030b330-e967-4317-bcd2-4664364ee362") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:35.826757 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.826724 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4zc\" (UniqueName: \"kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc\") pod \"network-check-target-hxfnv\" (UID: \"bf403608-4d7b-4585-88b9-7b1590654d3e\") " pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:35.826927 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:35.826768 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:35.826927 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.826920 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:35.827037 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.826942 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:35.827037 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.826954 2583 projected.go:196] Error preparing data for projected volume kube-api-access-fk4zc for pod openshift-network-diagnostics/network-check-target-hxfnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:35.827037 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.826963 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:35.827037 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.827012 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc podName:bf403608-4d7b-4585-88b9-7b1590654d3e nodeName:}" failed. No retries permitted until 2026-09-10 23:36:36.826996774 +0000 UTC m=+4.129474788 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fk4zc" (UniqueName: "kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc") pod "network-check-target-hxfnv" (UID: "bf403608-4d7b-4585-88b9-7b1590654d3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:35.827037 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:35.827029 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret podName:9b40d527-4121-4aeb-8abd-c4c7c5f53688 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:36.827021959 +0000 UTC m=+4.129499952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret") pod "global-pull-secret-syncer-rt86f" (UID: "9b40d527-4121-4aeb-8abd-c4c7c5f53688") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:36.166388 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:36.166313 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 23:31:35 +0000 UTC" deadline="2028-05-16 12:04:30.406412244 +0000 UTC" Sep 10 23:36:36.166388 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:36.166382 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14724h27m54.240035323s" Sep 10 23:36:36.449385 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:36.449354 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ca19af_6670_4b07_b1b3_d6737e89a1a4.slice/crio-ad188b275ee39d2cb975f20b89cac406da41956652682666c46b06464b8f5c5f WatchSource:0}: Error finding container ad188b275ee39d2cb975f20b89cac406da41956652682666c46b06464b8f5c5f: Status 404 returned error can't find the container with id ad188b275ee39d2cb975f20b89cac406da41956652682666c46b06464b8f5c5f Sep 10 23:36:36.451220 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:36.451188 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0170eaf4_0b83_4c76_be27_600a18f3226e.slice/crio-31395bfeefe552cb4bf06f01435dd3d0282c70112fb482bb4bde133a4695f616 WatchSource:0}: Error finding container 31395bfeefe552cb4bf06f01435dd3d0282c70112fb482bb4bde133a4695f616: Status 404 returned error can't find the container with id 31395bfeefe552cb4bf06f01435dd3d0282c70112fb482bb4bde133a4695f616 Sep 10 23:36:36.456360 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:36.456336 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0b07bf_513c_4c0f_aed5_bcfd96136254.slice/crio-07b56d0b9a83e483e89a35a76ff7d2b5c77d025593be1133043006b7b9b3fcdc WatchSource:0}: Error finding container 07b56d0b9a83e483e89a35a76ff7d2b5c77d025593be1133043006b7b9b3fcdc: Status 404 returned error can't find the container with id 07b56d0b9a83e483e89a35a76ff7d2b5c77d025593be1133043006b7b9b3fcdc Sep 10 23:36:36.459305 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:36.459283 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a3fa41_e1e7_481c_a52a_ffb83f4804cc.slice/crio-06d05202740999343f48aea3ef8c8fd7f2a740e1f2cd5cf1db84241a82932e02 WatchSource:0}: Error finding container 06d05202740999343f48aea3ef8c8fd7f2a740e1f2cd5cf1db84241a82932e02: Status 404 returned error can't find the container with id 06d05202740999343f48aea3ef8c8fd7f2a740e1f2cd5cf1db84241a82932e02 Sep 10 23:36:36.460965 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:36.460945 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f901d14_3c13_4ea6_8218_3653e4ef6ac7.slice/crio-8b3908162dc9708d989270acc3033629c630a1bee5da4a9f3af3faa93c21382d WatchSource:0}: Error finding container 8b3908162dc9708d989270acc3033629c630a1bee5da4a9f3af3faa93c21382d: Status 404 returned error can't find the container with id 8b3908162dc9708d989270acc3033629c630a1bee5da4a9f3af3faa93c21382d Sep 10 23:36:36.462935 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:36.462915 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f4ffdc_56fe_4eb6_a9ce_a48f4d1526c9.slice/crio-0d9779527ee154993bd2c248e2fa6b5952483be44b461250a6f668a0f4519d07 WatchSource:0}: Error finding container 0d9779527ee154993bd2c248e2fa6b5952483be44b461250a6f668a0f4519d07: Status 404 returned error can't find the container with id 0d9779527ee154993bd2c248e2fa6b5952483be44b461250a6f668a0f4519d07 Sep 10 23:36:36.464314 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:36.464293 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0331a4a5_abd1_419c_9116_60b56f90673b.slice/crio-7b6ee6641ccb3cf801d069ceed9aae7d642328caf979d4ce3875a57edb4f2565 WatchSource:0}: Error finding container 7b6ee6641ccb3cf801d069ceed9aae7d642328caf979d4ce3875a57edb4f2565: Status 404 returned error can't find the container with id 7b6ee6641ccb3cf801d069ceed9aae7d642328caf979d4ce3875a57edb4f2565 Sep 10 23:36:36.465455 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:36:36.465427 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05dad598_ac28_407d_b6c8_fa493d21c90f.slice/crio-1f2c7592cd986936532a2f6aa09947615baece8738bc5c47e5301d9782a9367c WatchSource:0}: Error finding container 1f2c7592cd986936532a2f6aa09947615baece8738bc5c47e5301d9782a9367c: Status 404 returned error can't find the container with id 1f2c7592cd986936532a2f6aa09947615baece8738bc5c47e5301d9782a9367c Sep 10 23:36:36.732223 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:36.732054 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs\") pod \"network-metrics-daemon-7hv8v\" (UID: \"e030b330-e967-4317-bcd2-4664364ee362\") " pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:36.732335 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:36.732268 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:36.732377 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:36.732338 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs podName:e030b330-e967-4317-bcd2-4664364ee362 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:38.732316537 +0000 UTC m=+6.034794533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs") pod "network-metrics-daemon-7hv8v" (UID: "e030b330-e967-4317-bcd2-4664364ee362") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:36.833505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:36.833471 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:36.833798 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:36.833557 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4zc\" (UniqueName: \"kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc\") pod \"network-check-target-hxfnv\" (UID: \"bf403608-4d7b-4585-88b9-7b1590654d3e\") " pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:36.833798 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:36.833669 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:36.833798 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:36.833706 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:36.833798 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:36.833728 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:36.833798 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:36.833742 2583 projected.go:196] Error preparing data for projected volume kube-api-access-fk4zc for pod openshift-network-diagnostics/network-check-target-hxfnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:36.833798 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:36.833753 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret podName:9b40d527-4121-4aeb-8abd-c4c7c5f53688 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:38.833732183 +0000 UTC m=+6.136210195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret") pod "global-pull-secret-syncer-rt86f" (UID: "9b40d527-4121-4aeb-8abd-c4c7c5f53688") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:36.834358 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:36.833806 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc podName:bf403608-4d7b-4585-88b9-7b1590654d3e nodeName:}" failed. No retries permitted until 2026-09-10 23:36:38.833788449 +0000 UTC m=+6.136266464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fk4zc" (UniqueName: "kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc") pod "network-check-target-hxfnv" (UID: "bf403608-4d7b-4585-88b9-7b1590654d3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:37.167392 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.167238 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 23:31:35 +0000 UTC" deadline="2028-05-30 19:58:47.972308714 +0000 UTC" Sep 10 23:36:37.167392 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.167277 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15068h22m10.805036878s" Sep 10 23:36:37.308330 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.308298 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:37.308522 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:37.308491 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:36:37.308620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.308605 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:37.308732 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:37.308715 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:36:37.309336 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.309313 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:37.309494 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:37.309471 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:36:37.336525 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.336489 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mwtvd" event={"ID":"31ca19af-6670-4b07-b1b3-d6737e89a1a4","Type":"ContainerStarted","Data":"ad188b275ee39d2cb975f20b89cac406da41956652682666c46b06464b8f5c5f"} Sep 10 23:36:37.343439 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.343197 2583 generic.go:356] "Generic (PLEG): container finished" podID="3779dff9bc8ed2f1165f8546a4a88973" containerID="a7b555e72830a0063ed8027e182659fa20b4168389f40a2c40958a48b06f0acb" exitCode=0 Sep 10 23:36:37.343439 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.343260 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal" event={"ID":"3779dff9bc8ed2f1165f8546a4a88973","Type":"ContainerDied","Data":"a7b555e72830a0063ed8027e182659fa20b4168389f40a2c40958a48b06f0acb"} Sep 10 23:36:37.356915 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.356835 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zlqzj" event={"ID":"3750ae5a-25be-4d56-89ae-67210d6cb357","Type":"ContainerStarted","Data":"9947171569e9370401ba3be3ea971712d51f3262937a27beb7a9af17f642cc3a"} Sep 10 23:36:37.370349 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.370299 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" event={"ID":"0331a4a5-abd1-419c-9116-60b56f90673b","Type":"ContainerStarted","Data":"7b6ee6641ccb3cf801d069ceed9aae7d642328caf979d4ce3875a57edb4f2565"} Sep 10 23:36:37.380634 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.380577 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pvxvx" event={"ID":"0170eaf4-0b83-4c76-be27-600a18f3226e","Type":"ContainerStarted","Data":"31395bfeefe552cb4bf06f01435dd3d0282c70112fb482bb4bde133a4695f616"} Sep 10 23:36:37.428851 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.428769 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-90.ec2.internal" event={"ID":"8aff0765baa894ad2af911b31f796a12","Type":"ContainerStarted","Data":"8f9e60e39c40fad2271f552b575c94047badf44d9602badbb3ade7b0f84dc4e7"} Sep 10 23:36:37.452166 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.452118 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" event={"ID":"05dad598-ac28-407d-b6c8-fa493d21c90f","Type":"ContainerStarted","Data":"1f2c7592cd986936532a2f6aa09947615baece8738bc5c47e5301d9782a9367c"} Sep 10 23:36:37.474780 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.474701 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gdmff" event={"ID":"79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9","Type":"ContainerStarted","Data":"0d9779527ee154993bd2c248e2fa6b5952483be44b461250a6f668a0f4519d07"} Sep 10 23:36:37.500713 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.500649 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" event={"ID":"2f901d14-3c13-4ea6-8218-3653e4ef6ac7","Type":"ContainerStarted","Data":"8b3908162dc9708d989270acc3033629c630a1bee5da4a9f3af3faa93c21382d"} Sep 10 23:36:37.508496 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.508278 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wnl97" event={"ID":"22a3fa41-e1e7-481c-a52a-ffb83f4804cc","Type":"ContainerStarted","Data":"06d05202740999343f48aea3ef8c8fd7f2a740e1f2cd5cf1db84241a82932e02"} Sep 10 23:36:37.532872 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:37.532836 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rvpbw" event={"ID":"1c0b07bf-513c-4c0f-aed5-bcfd96136254","Type":"ContainerStarted","Data":"07b56d0b9a83e483e89a35a76ff7d2b5c77d025593be1133043006b7b9b3fcdc"} Sep 10 23:36:38.569028 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:38.568439 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal" event={"ID":"3779dff9bc8ed2f1165f8546a4a88973","Type":"ContainerStarted","Data":"6752152012db9f7d63ec9f287c02d3b10ebb962f439af70ac4c36030ac35dba8"} Sep 10 23:36:38.599089 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:38.599029 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-90.ec2.internal" podStartSLOduration=4.599004353 podStartE2EDuration="4.599004353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:37.446349242 +0000 UTC m=+4.748827260" watchObservedRunningTime="2026-09-10 23:36:38.599004353 +0000 UTC m=+5.901482370" Sep 10 23:36:38.750086 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:38.750051 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs\") pod \"network-metrics-daemon-7hv8v\" (UID: \"e030b330-e967-4317-bcd2-4664364ee362\") " pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:38.750354 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:38.750278 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:38.750354 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:38.750343 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs podName:e030b330-e967-4317-bcd2-4664364ee362 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:42.75032325 +0000 UTC m=+10.052801247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs") pod "network-metrics-daemon-7hv8v" (UID: "e030b330-e967-4317-bcd2-4664364ee362") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:38.851158 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:38.851062 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:38.851334 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:38.851285 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:38.851394 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:38.851345 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret podName:9b40d527-4121-4aeb-8abd-c4c7c5f53688 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:42.851326569 +0000 UTC m=+10.153804575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret") pod "global-pull-secret-syncer-rt86f" (UID: "9b40d527-4121-4aeb-8abd-c4c7c5f53688") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:38.851459 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:38.851383 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:38.851459 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:38.851412 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:38.851459 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:38.851426 2583 projected.go:196] Error preparing data for projected volume kube-api-access-fk4zc for pod openshift-network-diagnostics/network-check-target-hxfnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:38.851621 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:38.851479 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc podName:bf403608-4d7b-4585-88b9-7b1590654d3e nodeName:}" failed. No retries permitted until 2026-09-10 23:36:42.851462164 +0000 UTC m=+10.153940161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fk4zc" (UniqueName: "kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc") pod "network-check-target-hxfnv" (UID: "bf403608-4d7b-4585-88b9-7b1590654d3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:38.851722 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:38.851698 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4zc\" (UniqueName: \"kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc\") pod \"network-check-target-hxfnv\" (UID: \"bf403608-4d7b-4585-88b9-7b1590654d3e\") " pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:39.305773 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:39.304917 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:39.305773 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:39.305051 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:36:39.305773 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:39.305457 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:39.305773 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:39.305562 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:36:39.305773 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:39.305609 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:39.305773 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:39.305691 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:36:41.307288 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:41.307258 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:41.307288 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:41.307280 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:41.307795 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:41.307397 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:36:41.307795 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:41.307462 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:36:41.307795 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:41.307528 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:41.307795 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:41.307592 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:36:42.784779 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:42.784741 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs\") pod \"network-metrics-daemon-7hv8v\" (UID: \"e030b330-e967-4317-bcd2-4664364ee362\") " pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:42.785275 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:42.784904 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:42.785275 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:42.784980 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs podName:e030b330-e967-4317-bcd2-4664364ee362 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:50.784956891 +0000 UTC m=+18.087434895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs") pod "network-metrics-daemon-7hv8v" (UID: "e030b330-e967-4317-bcd2-4664364ee362") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:42.885397 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:42.885363 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4zc\" (UniqueName: \"kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc\") pod \"network-check-target-hxfnv\" (UID: \"bf403608-4d7b-4585-88b9-7b1590654d3e\") " pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:42.885545 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:42.885429 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:42.885545 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:42.885513 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:42.885545 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:42.885539 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:42.885684 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:42.885552 2583 projected.go:196] Error preparing data for projected volume kube-api-access-fk4zc for pod openshift-network-diagnostics/network-check-target-hxfnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:42.885684 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:42.885560 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:42.885684 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:42.885608 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret podName:9b40d527-4121-4aeb-8abd-c4c7c5f53688 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:50.885589169 +0000 UTC m=+18.188067173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret") pod "global-pull-secret-syncer-rt86f" (UID: "9b40d527-4121-4aeb-8abd-c4c7c5f53688") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:42.885684 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:42.885624 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc podName:bf403608-4d7b-4585-88b9-7b1590654d3e nodeName:}" failed. No retries permitted until 2026-09-10 23:36:50.885616568 +0000 UTC m=+18.188094558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fk4zc" (UniqueName: "kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc") pod "network-check-target-hxfnv" (UID: "bf403608-4d7b-4585-88b9-7b1590654d3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:43.302197 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:43.302167 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:43.302358 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:43.302307 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:36:43.304657 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:43.304620 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:43.304773 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:43.304721 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:36:43.304773 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:43.304770 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:43.304901 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:43.304840 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:36:45.301868 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:45.301835 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:45.302329 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:45.301890 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:45.302329 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:45.301836 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:45.302329 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:45.301975 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:36:45.302329 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:45.302080 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:36:45.302329 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:45.302157 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:36:47.301833 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:47.301747 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:47.302305 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:47.301750 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:47.302305 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:47.301889 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:36:47.302305 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:47.301760 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:47.302305 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:47.301962 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:36:47.302305 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:47.302032 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:36:49.301759 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:49.301715 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:49.302200 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:49.301715 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:49.302200 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:49.301848 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:49.302200 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:49.301852 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:36:49.302200 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:49.301933 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:36:49.302200 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:49.302079 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:36:50.842981 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:50.842934 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs\") pod \"network-metrics-daemon-7hv8v\" (UID: \"e030b330-e967-4317-bcd2-4664364ee362\") " pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:50.843505 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:50.843124 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:50.843505 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:50.843205 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs podName:e030b330-e967-4317-bcd2-4664364ee362 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:06.843184539 +0000 UTC m=+34.145662547 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs") pod "network-metrics-daemon-7hv8v" (UID: "e030b330-e967-4317-bcd2-4664364ee362") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:50.943815 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:50.943775 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4zc\" (UniqueName: \"kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc\") pod \"network-check-target-hxfnv\" (UID: \"bf403608-4d7b-4585-88b9-7b1590654d3e\") " pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:50.943996 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:50.943830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:50.943996 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:50.943935 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:50.943996 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:50.943955 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:50.943996 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:50.943990 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:50.944232 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:50.944002 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret podName:9b40d527-4121-4aeb-8abd-c4c7c5f53688 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:06.943981951 +0000 UTC m=+34.246459943 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret") pod "global-pull-secret-syncer-rt86f" (UID: "9b40d527-4121-4aeb-8abd-c4c7c5f53688") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:50.944232 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:50.944003 2583 projected.go:196] Error preparing data for projected volume kube-api-access-fk4zc for pod openshift-network-diagnostics/network-check-target-hxfnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:50.944232 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:50.944048 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc podName:bf403608-4d7b-4585-88b9-7b1590654d3e nodeName:}" failed. No retries permitted until 2026-09-10 23:37:06.94403807 +0000 UTC m=+34.246516067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fk4zc" (UniqueName: "kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc") pod "network-check-target-hxfnv" (UID: "bf403608-4d7b-4585-88b9-7b1590654d3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:51.301643 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:51.301571 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:51.301643 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:51.301595 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:51.301863 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:51.301713 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:36:51.301863 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:51.301769 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:36:51.301863 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:51.301823 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:51.302007 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:51.301897 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:36:53.302453 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:53.302379 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:53.302914 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:53.302475 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:36:53.302914 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:53.302485 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:53.302914 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:53.302519 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:53.302914 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:53.302610 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:36:53.303119 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:53.302695 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:36:54.608979 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.608619 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" event={"ID":"0331a4a5-abd1-419c-9116-60b56f90673b","Type":"ContainerStarted","Data":"cc3329900e4c5f1156a02deeb9a546a91c7b34ecbf0a2c61bf0462a4cee0b881"} Sep 10 23:36:54.609824 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.609036 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" event={"ID":"0331a4a5-abd1-419c-9116-60b56f90673b","Type":"ContainerStarted","Data":"2a754db0423dc55de78eccd5d79480fd188a49d18e3434c154805ad9f6818bf4"} Sep 10 23:36:54.609824 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.609104 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" event={"ID":"0331a4a5-abd1-419c-9116-60b56f90673b","Type":"ContainerStarted","Data":"1810190dcd903f9522fa570bd5d55f0275c69bc110ccd1f865b274409628635d"} Sep 10 23:36:54.610544 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.610516 2583 generic.go:356] "Generic (PLEG): container finished" podID="0170eaf4-0b83-4c76-be27-600a18f3226e" containerID="813c5d059de299ebc835ab83b4f9de85febdd801a09a04e37060f7646a28811a" exitCode=0 Sep 10 23:36:54.610636 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.610605 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pvxvx" event={"ID":"0170eaf4-0b83-4c76-be27-600a18f3226e","Type":"ContainerDied","Data":"813c5d059de299ebc835ab83b4f9de85febdd801a09a04e37060f7646a28811a"} Sep 10 23:36:54.612579 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.612546 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" event={"ID":"05dad598-ac28-407d-b6c8-fa493d21c90f","Type":"ContainerStarted","Data":"46efa083726215e7077f26804693e13a826dd006bbe93fe112c09307ead6e369"} Sep 10 23:36:54.614340 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.614318 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gdmff" event={"ID":"79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9","Type":"ContainerStarted","Data":"171387ed12508c58061029ceba798e74f40076b43f34bada0b2be24b57732f6e"} Sep 10 23:36:54.619505 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.619420 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" event={"ID":"2f901d14-3c13-4ea6-8218-3653e4ef6ac7","Type":"ContainerStarted","Data":"64e24b2c4531e105e6957aeeaebde93f02b22998c26413ef33c42b5cc010deca"} Sep 10 23:36:54.620984 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.620964 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wnl97" event={"ID":"22a3fa41-e1e7-481c-a52a-ffb83f4804cc","Type":"ContainerStarted","Data":"79e73df5a4824f7fbdc308a03ee108994143cfdd10a6328002f612ad0236ff70"} Sep 10 23:36:54.626212 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.626185 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rvpbw" event={"ID":"1c0b07bf-513c-4c0f-aed5-bcfd96136254","Type":"ContainerStarted","Data":"960ee5c14337dba2e6ebadd6ae19f5b8c26c2838b907fc400c39b6c56e5956f3"} Sep 10 23:36:54.627875 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.627857 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zlqzj" event={"ID":"3750ae5a-25be-4d56-89ae-67210d6cb357","Type":"ContainerStarted","Data":"e1301d6654e9dfd0de650859c4e2480e3c0b11ee9710bc2efa6b6e93a159f5b0"} Sep 10 23:36:54.636512 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.636471 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-90.ec2.internal" podStartSLOduration=20.63645969 podStartE2EDuration="20.63645969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:38.599314181 +0000 UTC m=+5.901792199" watchObservedRunningTime="2026-09-10 23:36:54.63645969 +0000 UTC m=+21.938937702" Sep 10 23:36:54.648608 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.648568 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gdmff" podStartSLOduration=10.108366201 podStartE2EDuration="20.648555408s" totalImagesPullingTime="10.540189207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:54.647826637 +0000 UTC m=+21.950304648" watchObservedRunningTime="2026-09-10 23:36:54.648555408 +0000 UTC m=+21.951033420" Sep 10 23:36:54.660797 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.660762 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rvpbw" podStartSLOduration=2.892112133 podStartE2EDuration="20.660750695s" totalImagesPullingTime="17.768638562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:54.660652954 +0000 UTC m=+21.963130965" watchObservedRunningTime="2026-09-10 23:36:54.660750695 +0000 UTC m=+21.963228707" Sep 10 23:36:54.675808 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.675770 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kvg4b" podStartSLOduration=3.915585354 podStartE2EDuration="21.675757529s" totalImagesPullingTime="17.760172175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:54.675326561 +0000 UTC m=+21.977804573" watchObservedRunningTime="2026-09-10 23:36:54.675757529 +0000 UTC m=+21.978235540" Sep 10 23:36:54.691755 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.691722 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wnl97" podStartSLOduration=2.873658572 podStartE2EDuration="20.691708442s" totalImagesPullingTime="17.81804987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:54.691298979 +0000 UTC m=+21.993776991" watchObservedRunningTime="2026-09-10 23:36:54.691708442 +0000 UTC m=+21.994186453" Sep 10 23:36:54.705460 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:54.705429 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zlqzj" podStartSLOduration=2.950272599 podStartE2EDuration="20.705416942s" totalImagesPullingTime="17.755144343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:54.705207923 +0000 UTC m=+22.007685936" watchObservedRunningTime="2026-09-10 23:36:54.705416942 +0000 UTC m=+22.007894954" Sep 10 23:36:55.302269 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:55.302237 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:55.302436 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:55.302362 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:36:55.302544 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:55.302517 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:55.302681 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:55.302630 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:36:55.302681 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:55.302644 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:55.302807 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:55.302730 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:36:55.634820 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:55.633846 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" event={"ID":"0331a4a5-abd1-419c-9116-60b56f90673b","Type":"ContainerStarted","Data":"2b504eb3037606cd15bf800db9ac078419e9a2cb26421e52f9d86af4c2f87427"} Sep 10 23:36:55.634820 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:55.633936 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" event={"ID":"0331a4a5-abd1-419c-9116-60b56f90673b","Type":"ContainerStarted","Data":"0a1017a8ff7e9aaca437fd9948c43e6b834832b13152cea7ab7fb7048dae56a6"} Sep 10 23:36:55.634820 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:55.633986 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" event={"ID":"0331a4a5-abd1-419c-9116-60b56f90673b","Type":"ContainerStarted","Data":"b8455205a11666438d422173192aa68529ea5395b92afd3b257b41081142ddb9"} Sep 10 23:36:55.758037 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:55.758009 2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 23:36:56.164718 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:56.164566 2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T23:36:55.758032213Z","UUID":"729cba03-80f6-47f3-812d-cd195b8924c9","Handler":null,"Name":"","Endpoint":""} Sep 10 23:36:56.166271 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:56.166234 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 23:36:56.166271 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:56.166275 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 23:36:56.638654 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:56.638474 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" event={"ID":"2f901d14-3c13-4ea6-8218-3653e4ef6ac7","Type":"ContainerStarted","Data":"247c2916ebd6cb3435a3495da3295cea1e18fa7518b5c81336300387970c7ea9"} Sep 10 23:36:56.640469 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:56.640439 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mwtvd" event={"ID":"31ca19af-6670-4b07-b1b3-d6737e89a1a4","Type":"ContainerStarted","Data":"9b9eb98f102222664c79f2829a8b1b95c59e740f3d27c0d17532f002648a1b07"} Sep 10 23:36:56.654389 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:56.654348 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mwtvd" podStartSLOduration=5.879035634 podStartE2EDuration="23.654329651s" totalImagesPullingTime="17.775294017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:56.653830696 +0000 UTC m=+23.956308707" watchObservedRunningTime="2026-09-10 23:36:56.654329651 +0000 UTC m=+23.956807665" Sep 10 23:36:57.302240 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:57.302171 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:57.302376 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:57.302175 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:57.302376 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:57.302288 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:36:57.302376 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:57.302179 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:57.302538 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:57.302391 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:36:57.302538 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:57.302481 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:36:57.647291 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:57.647209 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" event={"ID":"0331a4a5-abd1-419c-9116-60b56f90673b","Type":"ContainerStarted","Data":"b34ea2bf3db6f65ee56c174d6780f4369dc92e33cd06e0ee50f726afb1c88baf"} Sep 10 23:36:57.649518 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:57.649482 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" event={"ID":"2f901d14-3c13-4ea6-8218-3653e4ef6ac7","Type":"ContainerStarted","Data":"55786b9c588dfb0d44d951316f359f4c486987c80d97e363c1b9a00af691a553"} Sep 10 23:36:57.669377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:57.669336 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxpv7" podStartSLOduration=4.804145055 podStartE2EDuration="24.669320964s" totalImagesPullingTime="19.865175909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:57.669291808 +0000 UTC m=+24.971769820" watchObservedRunningTime="2026-09-10 23:36:57.669320964 +0000 UTC m=+24.971798969" Sep 10 23:36:58.294159 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:58.293899 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gdmff" Sep 10 23:36:58.294508 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:58.294487 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gdmff" Sep 10 23:36:59.301492 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:59.301410 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:36:59.302379 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:59.301413 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:36:59.302379 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:59.301535 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:36:59.302379 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:59.301548 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:36:59.302379 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:59.301651 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:36:59.302379 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:36:59.301742 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:36:59.657805 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:59.657535 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" event={"ID":"0331a4a5-abd1-419c-9116-60b56f90673b","Type":"ContainerStarted","Data":"98a9ea6150b88cafcb0880a6c01e48a16a70610d05045825da3fd09a8974bd8b"} Sep 10 23:36:59.657805 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:59.657783 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:59.659652 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:59.659616 2583 generic.go:356] "Generic (PLEG): container finished" podID="0170eaf4-0b83-4c76-be27-600a18f3226e" containerID="7725e523ced2e90feab39d400105587ab8341cbc878bc5f92dede8fcb41c6c77" exitCode=0 Sep 10 23:36:59.659758 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:59.659665 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pvxvx" event={"ID":"0170eaf4-0b83-4c76-be27-600a18f3226e","Type":"ContainerDied","Data":"7725e523ced2e90feab39d400105587ab8341cbc878bc5f92dede8fcb41c6c77"} Sep 10 23:36:59.674633 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:59.674608 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:36:59.687230 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:36:59.687189 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" podStartSLOduration=7.862803795 podStartE2EDuration="25.687176182s" totalImagesPullingTime="17.824372387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:59.686470028 +0000 UTC m=+26.988948095" watchObservedRunningTime="2026-09-10 23:36:59.687176182 +0000 UTC m=+26.989654193" Sep 10 23:37:00.662512 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:00.662482 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:37:00.662919 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:00.662586 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:37:00.676933 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:00.676908 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:37:01.030091 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:01.030045 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rt86f"] Sep 10 23:37:01.030339 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:01.030191 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:37:01.030339 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:01.030295 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:37:01.032993 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:01.032967 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hxfnv"] Sep 10 23:37:01.033127 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:01.033093 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:37:01.033201 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:01.033177 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:37:01.033531 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:01.033510 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7hv8v"] Sep 10 23:37:01.033977 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:01.033603 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:37:01.033977 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:01.033698 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:37:01.666001 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:01.665971 2583 generic.go:356] "Generic (PLEG): container finished" podID="0170eaf4-0b83-4c76-be27-600a18f3226e" containerID="4038b4567caa1d6036336acd99f55710cec0b6c1906353be4c444064199957e8" exitCode=0 Sep 10 23:37:01.666368 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:01.666048 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pvxvx" event={"ID":"0170eaf4-0b83-4c76-be27-600a18f3226e","Type":"ContainerDied","Data":"4038b4567caa1d6036336acd99f55710cec0b6c1906353be4c444064199957e8"} Sep 10 23:37:01.845211 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:01.845176 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gdmff" Sep 10 23:37:01.845361 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:01.845315 2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 23:37:01.845719 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:01.845703 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gdmff" Sep 10 23:37:02.301805 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:02.301771 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:37:02.301959 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:02.301866 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:37:03.302354 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:03.302168 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:37:03.302746 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:03.302241 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:37:03.302746 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:03.302410 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:37:03.302746 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:03.302552 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:37:03.672116 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:03.672025 2583 generic.go:356] "Generic (PLEG): container finished" podID="0170eaf4-0b83-4c76-be27-600a18f3226e" containerID="5953fe3d2d818cba1fe78d789a6961881e0bbd67770934d51cacde9e2ce0da69" exitCode=0 Sep 10 23:37:03.672116 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:03.672086 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pvxvx" event={"ID":"0170eaf4-0b83-4c76-be27-600a18f3226e","Type":"ContainerDied","Data":"5953fe3d2d818cba1fe78d789a6961881e0bbd67770934d51cacde9e2ce0da69"} Sep 10 23:37:04.301823 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:04.301789 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:37:04.301979 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:04.301914 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:37:05.305670 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:05.305642 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:37:05.306290 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:05.305756 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hv8v" podUID="e030b330-e967-4317-bcd2-4664364ee362" Sep 10 23:37:05.306290 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:05.305823 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:37:05.306290 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:05.305907 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxfnv" podUID="bf403608-4d7b-4585-88b9-7b1590654d3e" Sep 10 23:37:06.302325 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:06.302286 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:37:06.302502 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:06.302429 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt86f" podUID="9b40d527-4121-4aeb-8abd-c4c7c5f53688" Sep 10 23:37:06.882344 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:06.882301 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs\") pod \"network-metrics-daemon-7hv8v\" (UID: \"e030b330-e967-4317-bcd2-4664364ee362\") " pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:37:06.882844 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:06.882472 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:37:06.882844 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:06.882551 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs podName:e030b330-e967-4317-bcd2-4664364ee362 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:38.882529553 +0000 UTC m=+66.185007565 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs") pod "network-metrics-daemon-7hv8v" (UID: "e030b330-e967-4317-bcd2-4664364ee362") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:37:06.983258 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:06.983222 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4zc\" (UniqueName: \"kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc\") pod \"network-check-target-hxfnv\" (UID: \"bf403608-4d7b-4585-88b9-7b1590654d3e\") " pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:37:06.983414 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:06.983279 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:37:06.983414 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:06.983366 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:37:06.983414 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:06.983392 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:37:06.983414 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:06.983404 2583 projected.go:196] Error preparing data for projected volume kube-api-access-fk4zc for pod openshift-network-diagnostics/network-check-target-hxfnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:37:06.983580 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:06.983425 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:37:06.983580 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:06.983453 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc podName:bf403608-4d7b-4585-88b9-7b1590654d3e nodeName:}" failed. No retries permitted until 2026-09-10 23:37:38.983438047 +0000 UTC m=+66.285916050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fk4zc" (UniqueName: "kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc") pod "network-check-target-hxfnv" (UID: "bf403608-4d7b-4585-88b9-7b1590654d3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:37:06.983580 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:06.983472 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret podName:9b40d527-4121-4aeb-8abd-c4c7c5f53688 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:38.983458925 +0000 UTC m=+66.285936915 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret") pod "global-pull-secret-syncer-rt86f" (UID: "9b40d527-4121-4aeb-8abd-c4c7c5f53688") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:37:07.055082 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.054997 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-90.ec2.internal" event="NodeReady" Sep 10 23:37:07.055229 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.055166 2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 23:37:07.103303 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.103268 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lfxmq"] Sep 10 23:37:07.108743 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.108720 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xlw45"] Sep 10 23:37:07.108892 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.108861 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:07.111784 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.111763 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 23:37:07.111912 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.111764 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 23:37:07.111912 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.111796 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9kf9v\"" Sep 10 23:37:07.112386 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.112368 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xlw45" Sep 10 23:37:07.114711 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.114679 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 23:37:07.115154 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.115131 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 23:37:07.115240 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.115136 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 23:37:07.115570 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.115549 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-njwmd\"" Sep 10 23:37:07.116757 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.115989 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xlw45"] Sep 10 23:37:07.116861 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.116842 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lfxmq"] Sep 10 23:37:07.284983 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.284950 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08605f38-1013-4f84-b980-10bada8fa720-config-volume\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:07.284983 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.284985 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqw5p\" (UniqueName: \"kubernetes.io/projected/08605f38-1013-4f84-b980-10bada8fa720-kube-api-access-gqw5p\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:07.285249 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.285003 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08605f38-1013-4f84-b980-10bada8fa720-tmp-dir\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:07.285249 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.285020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56489c59-0fc9-4469-a76d-141755000565-cert\") pod \"ingress-canary-xlw45\" (UID: \"56489c59-0fc9-4469-a76d-141755000565\") " pod="openshift-ingress-canary/ingress-canary-xlw45" Sep 10 23:37:07.285249 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.285188 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6sxw\" (UniqueName: \"kubernetes.io/projected/56489c59-0fc9-4469-a76d-141755000565-kube-api-access-p6sxw\") pod \"ingress-canary-xlw45\" (UID: \"56489c59-0fc9-4469-a76d-141755000565\") " pod="openshift-ingress-canary/ingress-canary-xlw45" Sep 10 23:37:07.285249 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.285230 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:07.308210 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.308141 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:37:07.308360 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.308141 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:37:07.310859 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.310830 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 23:37:07.310998 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.310867 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 23:37:07.310998 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.310934 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 23:37:07.310998 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.310830 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t4h6l\"" Sep 10 23:37:07.311162 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.311095 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xmh4\"" Sep 10 23:37:07.385797 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.385762 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56489c59-0fc9-4469-a76d-141755000565-cert\") pod \"ingress-canary-xlw45\" (UID: \"56489c59-0fc9-4469-a76d-141755000565\") " pod="openshift-ingress-canary/ingress-canary-xlw45" Sep 10 23:37:07.385975 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.385858 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6sxw\" (UniqueName: \"kubernetes.io/projected/56489c59-0fc9-4469-a76d-141755000565-kube-api-access-p6sxw\") pod \"ingress-canary-xlw45\" (UID: \"56489c59-0fc9-4469-a76d-141755000565\") " pod="openshift-ingress-canary/ingress-canary-xlw45" Sep 10 23:37:07.385975 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.385885 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:07.385975 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.385937 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08605f38-1013-4f84-b980-10bada8fa720-config-volume\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:07.385975 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.385966 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqw5p\" (UniqueName: \"kubernetes.io/projected/08605f38-1013-4f84-b980-10bada8fa720-kube-api-access-gqw5p\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:07.386167 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.385991 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08605f38-1013-4f84-b980-10bada8fa720-tmp-dir\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:07.386167 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:07.386037 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 23:37:07.386167 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:07.386127 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls podName:08605f38-1013-4f84-b980-10bada8fa720 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:07.886105889 +0000 UTC m=+35.188583894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls") pod "dns-default-lfxmq" (UID: "08605f38-1013-4f84-b980-10bada8fa720") : secret "dns-default-metrics-tls" not found Sep 10 23:37:07.386339 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.386321 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08605f38-1013-4f84-b980-10bada8fa720-tmp-dir\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:07.386814 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.386788 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08605f38-1013-4f84-b980-10bada8fa720-config-volume\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:07.390836 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.390816 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56489c59-0fc9-4469-a76d-141755000565-cert\") pod \"ingress-canary-xlw45\" (UID: \"56489c59-0fc9-4469-a76d-141755000565\") " pod="openshift-ingress-canary/ingress-canary-xlw45" Sep 10 23:37:07.394291 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.394263 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqw5p\" (UniqueName: \"kubernetes.io/projected/08605f38-1013-4f84-b980-10bada8fa720-kube-api-access-gqw5p\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:07.394465 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.394442 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6sxw\" (UniqueName: \"kubernetes.io/projected/56489c59-0fc9-4469-a76d-141755000565-kube-api-access-p6sxw\") pod \"ingress-canary-xlw45\" (UID: \"56489c59-0fc9-4469-a76d-141755000565\") " pod="openshift-ingress-canary/ingress-canary-xlw45" Sep 10 23:37:07.427418 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.427383 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xlw45" Sep 10 23:37:07.686013 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.685830 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pvxvx" event={"ID":"0170eaf4-0b83-4c76-be27-600a18f3226e","Type":"ContainerStarted","Data":"43f5f82ae339435f13f86eec81245b72485f7fea9d2880d109c1cef9093952c7"} Sep 10 23:37:07.737557 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.737530 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xlw45"] Sep 10 23:37:07.767956 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:37:07.767923 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56489c59_0fc9_4469_a76d_141755000565.slice/crio-53e6a1e177466dc8815d302b5e32af720d48c4c87510d000b192fc81773efd3d WatchSource:0}: Error finding container 53e6a1e177466dc8815d302b5e32af720d48c4c87510d000b192fc81773efd3d: Status 404 returned error can't find the container with id 53e6a1e177466dc8815d302b5e32af720d48c4c87510d000b192fc81773efd3d Sep 10 23:37:07.890241 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:07.890158 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:07.890644 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:07.890276 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 23:37:07.890644 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:07.890335 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls podName:08605f38-1013-4f84-b980-10bada8fa720 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:08.890316675 +0000 UTC m=+36.192794670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls") pod "dns-default-lfxmq" (UID: "08605f38-1013-4f84-b980-10bada8fa720") : secret "dns-default-metrics-tls" not found Sep 10 23:37:08.302032 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:08.301951 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:37:08.304513 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:08.304490 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 23:37:08.693190 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:08.693095 2583 generic.go:356] "Generic (PLEG): container finished" podID="0170eaf4-0b83-4c76-be27-600a18f3226e" containerID="43f5f82ae339435f13f86eec81245b72485f7fea9d2880d109c1cef9093952c7" exitCode=0 Sep 10 23:37:08.693190 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:08.693131 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pvxvx" event={"ID":"0170eaf4-0b83-4c76-be27-600a18f3226e","Type":"ContainerDied","Data":"43f5f82ae339435f13f86eec81245b72485f7fea9d2880d109c1cef9093952c7"} Sep 10 23:37:08.694622 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:08.694598 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xlw45" event={"ID":"56489c59-0fc9-4469-a76d-141755000565","Type":"ContainerStarted","Data":"53e6a1e177466dc8815d302b5e32af720d48c4c87510d000b192fc81773efd3d"} Sep 10 23:37:08.898537 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:08.898508 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:08.899204 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:08.898703 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 23:37:08.899204 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:08.898796 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls podName:08605f38-1013-4f84-b980-10bada8fa720 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:10.898755622 +0000 UTC m=+38.201233627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls") pod "dns-default-lfxmq" (UID: "08605f38-1013-4f84-b980-10bada8fa720") : secret "dns-default-metrics-tls" not found Sep 10 23:37:09.701205 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:09.701166 2583 generic.go:356] "Generic (PLEG): container finished" podID="0170eaf4-0b83-4c76-be27-600a18f3226e" containerID="e9a1994f6c52b5b97ffb4cda48ed0bb7644c96668a4a9a1b3395ba5d307ea1df" exitCode=0 Sep 10 23:37:09.701379 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:09.701226 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pvxvx" event={"ID":"0170eaf4-0b83-4c76-be27-600a18f3226e","Type":"ContainerDied","Data":"e9a1994f6c52b5b97ffb4cda48ed0bb7644c96668a4a9a1b3395ba5d307ea1df"} Sep 10 23:37:09.869227 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:09.869196 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bvsht"] Sep 10 23:37:09.872188 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:09.872172 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bvsht" Sep 10 23:37:09.874741 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:09.874723 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 10 23:37:09.874844 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:09.874768 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mfmg7\"" Sep 10 23:37:09.874844 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:09.874795 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 10 23:37:09.881387 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:09.881364 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bvsht"] Sep 10 23:37:10.007041 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.006887 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j74km\" (UniqueName: \"kubernetes.io/projected/ba502de5-22a2-417b-9c16-e281af5cd43a-kube-api-access-j74km\") pod \"migrator-8b4d697c5-bvsht\" (UID: \"ba502de5-22a2-417b-9c16-e281af5cd43a\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bvsht" Sep 10 23:37:10.107463 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.107427 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j74km\" (UniqueName: \"kubernetes.io/projected/ba502de5-22a2-417b-9c16-e281af5cd43a-kube-api-access-j74km\") pod \"migrator-8b4d697c5-bvsht\" (UID: \"ba502de5-22a2-417b-9c16-e281af5cd43a\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bvsht" Sep 10 23:37:10.117573 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.117540 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j74km\" (UniqueName: \"kubernetes.io/projected/ba502de5-22a2-417b-9c16-e281af5cd43a-kube-api-access-j74km\") pod \"migrator-8b4d697c5-bvsht\" (UID: \"ba502de5-22a2-417b-9c16-e281af5cd43a\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bvsht" Sep 10 23:37:10.182644 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.182606 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bvsht" Sep 10 23:37:10.451561 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.451532 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-bvsht"] Sep 10 23:37:10.454666 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:37:10.454644 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba502de5_22a2_417b_9c16_e281af5cd43a.slice/crio-1226d84aed17a4cf2c4d8702d82574f4630e44ac87ff56dafe28c54607db1842 WatchSource:0}: Error finding container 1226d84aed17a4cf2c4d8702d82574f4630e44ac87ff56dafe28c54607db1842: Status 404 returned error can't find the container with id 1226d84aed17a4cf2c4d8702d82574f4630e44ac87ff56dafe28c54607db1842 Sep 10 23:37:10.589547 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.589524 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-n5hcp"] Sep 10 23:37:10.593645 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.593630 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" Sep 10 23:37:10.596276 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.596249 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 10 23:37:10.596421 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.596339 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 10 23:37:10.596421 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.596403 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mzm5m\"" Sep 10 23:37:10.596541 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.596479 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 10 23:37:10.596612 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.596597 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 10 23:37:10.603420 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.603402 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-n5hcp"] Sep 10 23:37:10.705105 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.705057 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bvsht" event={"ID":"ba502de5-22a2-417b-9c16-e281af5cd43a","Type":"ContainerStarted","Data":"1226d84aed17a4cf2c4d8702d82574f4630e44ac87ff56dafe28c54607db1842"} Sep 10 23:37:10.708178 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.708156 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pvxvx" event={"ID":"0170eaf4-0b83-4c76-be27-600a18f3226e","Type":"ContainerStarted","Data":"a2d0bc3ae00daf461f853bd60eb7e5a7562197eca3c2c0a99ec6d5b5211adee4"} Sep 10 23:37:10.709564 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.709543 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xlw45" event={"ID":"56489c59-0fc9-4469-a76d-141755000565","Type":"ContainerStarted","Data":"1c2007bef751a9c2066216883860a3285d47e0e671b6026a98539ffecf5ebddf"} Sep 10 23:37:10.712487 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.712469 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0bd979c-1dda-4b39-a5c0-6b7d1797bee8-signing-key\") pod \"service-ca-568869fdc5-n5hcp\" (UID: \"e0bd979c-1dda-4b39-a5c0-6b7d1797bee8\") " pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" Sep 10 23:37:10.712551 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.712510 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0bd979c-1dda-4b39-a5c0-6b7d1797bee8-signing-cabundle\") pod \"service-ca-568869fdc5-n5hcp\" (UID: \"e0bd979c-1dda-4b39-a5c0-6b7d1797bee8\") " pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" Sep 10 23:37:10.712584 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.712567 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c85j5\" (UniqueName: \"kubernetes.io/projected/e0bd979c-1dda-4b39-a5c0-6b7d1797bee8-kube-api-access-c85j5\") pod \"service-ca-568869fdc5-n5hcp\" (UID: \"e0bd979c-1dda-4b39-a5c0-6b7d1797bee8\") " pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" Sep 10 23:37:10.731192 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.731149 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pvxvx" podStartSLOduration=8.328545655 podStartE2EDuration="36.731135645s" totalImagesPullingTime="28.40258999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:10.730588198 +0000 UTC m=+38.033066209" watchObservedRunningTime="2026-09-10 23:37:10.731135645 +0000 UTC m=+38.033613657" Sep 10 23:37:10.750593 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.750534 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xlw45" podStartSLOduration=1.5913410080000001 podStartE2EDuration="3.750520558s" totalImagesPullingTime="2.15917955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:10.750117286 +0000 UTC m=+38.052595298" watchObservedRunningTime="2026-09-10 23:37:10.750520558 +0000 UTC m=+38.052998569" Sep 10 23:37:10.812865 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.812838 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c85j5\" (UniqueName: \"kubernetes.io/projected/e0bd979c-1dda-4b39-a5c0-6b7d1797bee8-kube-api-access-c85j5\") pod \"service-ca-568869fdc5-n5hcp\" (UID: \"e0bd979c-1dda-4b39-a5c0-6b7d1797bee8\") " pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" Sep 10 23:37:10.813022 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.812896 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0bd979c-1dda-4b39-a5c0-6b7d1797bee8-signing-key\") pod \"service-ca-568869fdc5-n5hcp\" (UID: \"e0bd979c-1dda-4b39-a5c0-6b7d1797bee8\") " pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" Sep 10 23:37:10.813273 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.813251 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0bd979c-1dda-4b39-a5c0-6b7d1797bee8-signing-cabundle\") pod \"service-ca-568869fdc5-n5hcp\" (UID: \"e0bd979c-1dda-4b39-a5c0-6b7d1797bee8\") " pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" Sep 10 23:37:10.813936 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.813903 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0bd979c-1dda-4b39-a5c0-6b7d1797bee8-signing-cabundle\") pod \"service-ca-568869fdc5-n5hcp\" (UID: \"e0bd979c-1dda-4b39-a5c0-6b7d1797bee8\") " pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" Sep 10 23:37:10.815642 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.815622 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0bd979c-1dda-4b39-a5c0-6b7d1797bee8-signing-key\") pod \"service-ca-568869fdc5-n5hcp\" (UID: \"e0bd979c-1dda-4b39-a5c0-6b7d1797bee8\") " pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" Sep 10 23:37:10.827249 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.827223 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c85j5\" (UniqueName: \"kubernetes.io/projected/e0bd979c-1dda-4b39-a5c0-6b7d1797bee8-kube-api-access-c85j5\") pod \"service-ca-568869fdc5-n5hcp\" (UID: \"e0bd979c-1dda-4b39-a5c0-6b7d1797bee8\") " pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" Sep 10 23:37:10.886183 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.886126 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7s59l"] Sep 10 23:37:10.889442 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.889424 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:10.891808 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.891784 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 23:37:10.891922 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.891903 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 23:37:10.891981 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.891964 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 23:37:10.892042 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.891999 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 23:37:10.892255 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.892239 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-424q2\"" Sep 10 23:37:10.902427 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.902406 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" Sep 10 23:37:10.903315 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.903295 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7s59l"] Sep 10 23:37:10.915247 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:10.915220 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:10.915518 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:10.915495 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 23:37:10.915627 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:10.915567 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls podName:08605f38-1013-4f84-b980-10bada8fa720 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:14.91554481 +0000 UTC m=+42.218022805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls") pod "dns-default-lfxmq" (UID: "08605f38-1013-4f84-b980-10bada8fa720") : secret "dns-default-metrics-tls" not found Sep 10 23:37:11.015940 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.015761 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-data-volume\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.016537 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.015950 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.016537 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.016005 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.016537 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.016025 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzj5\" (UniqueName: \"kubernetes.io/projected/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-api-access-kbzj5\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.016537 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.016056 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-crio-socket\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.116635 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.116591 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.116635 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.116638 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzj5\" (UniqueName: \"kubernetes.io/projected/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-api-access-kbzj5\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.116847 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:11.116767 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 23:37:11.116847 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.116814 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-crio-socket\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.116941 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:11.116857 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls podName:3c94d0ce-50fc-4170-b7ad-87648ed95d4f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:11.616834156 +0000 UTC m=+38.919312150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7s59l" (UID: "3c94d0ce-50fc-4170-b7ad-87648ed95d4f") : secret "insights-runtime-extractor-tls" not found Sep 10 23:37:11.116941 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.116935 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-data-volume\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.117243 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.116960 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.117243 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.117032 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-crio-socket\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.117243 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:11.117042 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 23:37:11.117243 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:11.117109 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm podName:3c94d0ce-50fc-4170-b7ad-87648ed95d4f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:11.617094348 +0000 UTC m=+38.919572340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7s59l" (UID: "3c94d0ce-50fc-4170-b7ad-87648ed95d4f") : configmap "kube-rbac-proxy" not found Sep 10 23:37:11.117544 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.117347 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-data-volume\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.127224 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.127199 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzj5\" (UniqueName: \"kubernetes.io/projected/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-api-access-kbzj5\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.181459 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.181393 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-n5hcp"] Sep 10 23:37:11.185228 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:37:11.185200 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bd979c_1dda_4b39_a5c0_6b7d1797bee8.slice/crio-ca2d19f8cd9ce94dff46c399f8a396334e43fdf81b0b8c868c368eb5cb78d654 WatchSource:0}: Error finding container ca2d19f8cd9ce94dff46c399f8a396334e43fdf81b0b8c868c368eb5cb78d654: Status 404 returned error can't find the container with id ca2d19f8cd9ce94dff46c399f8a396334e43fdf81b0b8c868c368eb5cb78d654 Sep 10 23:37:11.620830 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.620746 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.620975 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.620834 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:11.620975 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:11.620879 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 23:37:11.620975 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:11.620935 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 23:37:11.620975 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:11.620957 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm podName:3c94d0ce-50fc-4170-b7ad-87648ed95d4f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:12.620935877 +0000 UTC m=+39.923413879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7s59l" (UID: "3c94d0ce-50fc-4170-b7ad-87648ed95d4f") : configmap "kube-rbac-proxy" not found Sep 10 23:37:11.620975 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:11.620977 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls podName:3c94d0ce-50fc-4170-b7ad-87648ed95d4f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:12.620965286 +0000 UTC m=+39.923443281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7s59l" (UID: "3c94d0ce-50fc-4170-b7ad-87648ed95d4f") : secret "insights-runtime-extractor-tls" not found Sep 10 23:37:11.713295 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:11.713247 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" event={"ID":"e0bd979c-1dda-4b39-a5c0-6b7d1797bee8","Type":"ContainerStarted","Data":"ca2d19f8cd9ce94dff46c399f8a396334e43fdf81b0b8c868c368eb5cb78d654"} Sep 10 23:37:12.627658 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:12.627563 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:12.627658 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:12.627642 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:12.628194 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:12.627697 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 23:37:12.628194 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:12.627765 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm podName:3c94d0ce-50fc-4170-b7ad-87648ed95d4f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:14.627746136 +0000 UTC m=+41.930224152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7s59l" (UID: "3c94d0ce-50fc-4170-b7ad-87648ed95d4f") : configmap "kube-rbac-proxy" not found Sep 10 23:37:12.628194 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:12.627766 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 23:37:12.628194 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:12.627803 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls podName:3c94d0ce-50fc-4170-b7ad-87648ed95d4f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:14.627793035 +0000 UTC m=+41.930271039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7s59l" (UID: "3c94d0ce-50fc-4170-b7ad-87648ed95d4f") : secret "insights-runtime-extractor-tls" not found Sep 10 23:37:12.717924 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:12.717888 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bvsht" event={"ID":"ba502de5-22a2-417b-9c16-e281af5cd43a","Type":"ContainerStarted","Data":"dc691ac70aca173e88fb1c46938a207ce45d92aaee6a0b0e9e8e1d97b7f76dd5"} Sep 10 23:37:12.718127 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:12.717943 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bvsht" event={"ID":"ba502de5-22a2-417b-9c16-e281af5cd43a","Type":"ContainerStarted","Data":"8db511fd567bbf21cac21f7ba20898250a72363d644275d7c77fbb19a48b0c83"} Sep 10 23:37:12.734835 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:12.734782 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-bvsht" podStartSLOduration=1.930418449 podStartE2EDuration="3.734756846s" totalImagesPullingTime="1.804338397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:12.733416259 +0000 UTC m=+40.035894465" watchObservedRunningTime="2026-09-10 23:37:12.734756846 +0000 UTC m=+40.037234858" Sep 10 23:37:13.722126 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:13.722032 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" event={"ID":"e0bd979c-1dda-4b39-a5c0-6b7d1797bee8","Type":"ContainerStarted","Data":"ca9ae71c76ffcbf5fab0e9a4d26815b94bba8899964936dbed04829379e3bf28"} Sep 10 23:37:13.739681 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:13.739632 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-n5hcp" podStartSLOduration=1.643283628 podStartE2EDuration="3.739617427s" totalImagesPullingTime="2.096333799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:13.738915424 +0000 UTC m=+41.041393436" watchObservedRunningTime="2026-09-10 23:37:13.739617427 +0000 UTC m=+41.042095438" Sep 10 23:37:13.824944 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:13.824918 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rvpbw_1c0b07bf-513c-4c0f-aed5-bcfd96136254/dns-node-resolver/0.log" Sep 10 23:37:14.622449 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:14.622417 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zlqzj_3750ae5a-25be-4d56-89ae-67210d6cb357/node-ca/0.log" Sep 10 23:37:14.643782 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:14.643748 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:14.644001 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:14.643827 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:14.644001 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:14.643913 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 23:37:14.644001 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:14.643944 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 23:37:14.644001 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:14.644000 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm podName:3c94d0ce-50fc-4170-b7ad-87648ed95d4f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:18.643977588 +0000 UTC m=+45.946455593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7s59l" (UID: "3c94d0ce-50fc-4170-b7ad-87648ed95d4f") : configmap "kube-rbac-proxy" not found Sep 10 23:37:14.644230 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:14.644020 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls podName:3c94d0ce-50fc-4170-b7ad-87648ed95d4f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:18.644010189 +0000 UTC m=+45.946488182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7s59l" (UID: "3c94d0ce-50fc-4170-b7ad-87648ed95d4f") : secret "insights-runtime-extractor-tls" not found Sep 10 23:37:14.946657 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:14.946557 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:14.947115 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:14.946721 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 23:37:14.947115 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:14.946829 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls podName:08605f38-1013-4f84-b980-10bada8fa720 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:22.946784824 +0000 UTC m=+50.249262821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls") pod "dns-default-lfxmq" (UID: "08605f38-1013-4f84-b980-10bada8fa720") : secret "dns-default-metrics-tls" not found Sep 10 23:37:15.621105 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:15.621054 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xlw45_56489c59-0fc9-4469-a76d-141755000565/serve-healthcheck-canary/0.log" Sep 10 23:37:15.825858 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:15.825831 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bvsht_ba502de5-22a2-417b-9c16-e281af5cd43a/migrator/0.log" Sep 10 23:37:16.021368 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:16.021284 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bvsht_ba502de5-22a2-417b-9c16-e281af5cd43a/graceful-termination/0.log" Sep 10 23:37:18.676488 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:18.676453 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:18.676978 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:18.676522 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:18.676978 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:18.676593 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 23:37:18.676978 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:18.676653 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm podName:3c94d0ce-50fc-4170-b7ad-87648ed95d4f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:26.676635798 +0000 UTC m=+53.979113795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7s59l" (UID: "3c94d0ce-50fc-4170-b7ad-87648ed95d4f") : configmap "kube-rbac-proxy" not found Sep 10 23:37:18.676978 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:18.676656 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 23:37:18.676978 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:37:18.676721 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls podName:3c94d0ce-50fc-4170-b7ad-87648ed95d4f nodeName:}" failed. No retries permitted until 2026-09-10 23:37:26.676700727 +0000 UTC m=+53.979178724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7s59l" (UID: "3c94d0ce-50fc-4170-b7ad-87648ed95d4f") : secret "insights-runtime-extractor-tls" not found Sep 10 23:37:23.009937 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:23.009897 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:23.012662 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:23.012640 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08605f38-1013-4f84-b980-10bada8fa720-metrics-tls\") pod \"dns-default-lfxmq\" (UID: \"08605f38-1013-4f84-b980-10bada8fa720\") " pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:23.021706 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:23.021680 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:23.323849 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:23.323773 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lfxmq"] Sep 10 23:37:23.326655 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:37:23.326632 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08605f38_1013_4f84_b980_10bada8fa720.slice/crio-081b0c574aa293a5e9ef4598aca85e62146bbcd485ae3f238813056c5de6b2ca WatchSource:0}: Error finding container 081b0c574aa293a5e9ef4598aca85e62146bbcd485ae3f238813056c5de6b2ca: Status 404 returned error can't find the container with id 081b0c574aa293a5e9ef4598aca85e62146bbcd485ae3f238813056c5de6b2ca Sep 10 23:37:23.751324 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:23.751290 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lfxmq" event={"ID":"08605f38-1013-4f84-b980-10bada8fa720","Type":"ContainerStarted","Data":"081b0c574aa293a5e9ef4598aca85e62146bbcd485ae3f238813056c5de6b2ca"} Sep 10 23:37:25.758080 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:25.758034 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lfxmq" event={"ID":"08605f38-1013-4f84-b980-10bada8fa720","Type":"ContainerStarted","Data":"f10d13b6b90b137e73be206c66d6b4866fc1af32551430ed5a69468eaac9fd4c"} Sep 10 23:37:26.738744 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:26.738696 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:26.738997 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:26.738805 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:26.739358 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:26.739339 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:26.741147 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:26.741124 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c94d0ce-50fc-4170-b7ad-87648ed95d4f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7s59l\" (UID: \"3c94d0ce-50fc-4170-b7ad-87648ed95d4f\") " pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:26.763318 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:26.763287 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lfxmq" event={"ID":"08605f38-1013-4f84-b980-10bada8fa720","Type":"ContainerStarted","Data":"1c0f0b832a73ab2e7cab158705f2fac841769ea01d11c8d8f3613f31154c79d7"} Sep 10 23:37:26.763716 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:26.763413 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:26.779937 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:26.779889 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lfxmq" podStartSLOduration=17.503550521 podStartE2EDuration="19.7798759s" totalImagesPullingTime="2.276325379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:26.778643042 +0000 UTC m=+54.081121045" watchObservedRunningTime="2026-09-10 23:37:26.7798759 +0000 UTC m=+54.082353905" Sep 10 23:37:26.799257 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:26.799234 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7s59l" Sep 10 23:37:27.068411 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:27.068335 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7s59l"] Sep 10 23:37:27.070896 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:37:27.070871 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c94d0ce_50fc_4170_b7ad_87648ed95d4f.slice/crio-a3c56ccc30ff6d46f7cacbcb9bfa1636b53f6fc100fbdee1f30754e1844064cd WatchSource:0}: Error finding container a3c56ccc30ff6d46f7cacbcb9bfa1636b53f6fc100fbdee1f30754e1844064cd: Status 404 returned error can't find the container with id a3c56ccc30ff6d46f7cacbcb9bfa1636b53f6fc100fbdee1f30754e1844064cd Sep 10 23:37:27.768613 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:27.768573 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7s59l" event={"ID":"3c94d0ce-50fc-4170-b7ad-87648ed95d4f","Type":"ContainerStarted","Data":"2da209696010e775e79491950901a2754cbc99dc59aac78cc44a5c60e0b64c84"} Sep 10 23:37:27.769051 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:27.768642 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7s59l" event={"ID":"3c94d0ce-50fc-4170-b7ad-87648ed95d4f","Type":"ContainerStarted","Data":"a3c56ccc30ff6d46f7cacbcb9bfa1636b53f6fc100fbdee1f30754e1844064cd"} Sep 10 23:37:28.774045 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:28.774005 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7s59l" event={"ID":"3c94d0ce-50fc-4170-b7ad-87648ed95d4f","Type":"ContainerStarted","Data":"fd67e95be221a76329a191793b9072de5f0bf07510c162f2e2e88ae915159d8b"} Sep 10 23:37:29.262923 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.262899 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm"] Sep 10 23:37:29.265825 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.265806 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q"] Sep 10 23:37:29.265964 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.265917 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm" Sep 10 23:37:29.268518 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.268499 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.268676 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.268577 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 23:37:29.269214 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.269199 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 23:37:29.269511 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.269492 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 10 23:37:29.269628 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.269522 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 23:37:29.269628 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.269497 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-v9rsr\"" Sep 10 23:37:29.270926 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.270908 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 10 23:37:29.271153 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.271133 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 10 23:37:29.271244 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.271163 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 10 23:37:29.271301 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.271285 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 10 23:37:29.278126 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.278105 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm"] Sep 10 23:37:29.281952 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.281933 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q"] Sep 10 23:37:29.355449 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.355425 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25316150-6e7e-45ab-8d43-2bcf46ac51cb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-755bc595fd-j2wkm\" (UID: \"25316150-6e7e-45ab-8d43-2bcf46ac51cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm" Sep 10 23:37:29.355567 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.355467 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-hub\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.355567 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.355488 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.355567 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.355532 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjsj\" (UniqueName: \"kubernetes.io/projected/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-kube-api-access-lkjsj\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.355567 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.355565 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.355696 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.355595 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-ca\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.355696 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.355671 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp64v\" (UniqueName: \"kubernetes.io/projected/25316150-6e7e-45ab-8d43-2bcf46ac51cb-kube-api-access-cp64v\") pod \"managed-serviceaccount-addon-agent-755bc595fd-j2wkm\" (UID: \"25316150-6e7e-45ab-8d43-2bcf46ac51cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm" Sep 10 23:37:29.355757 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.355694 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.456201 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.456171 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp64v\" (UniqueName: \"kubernetes.io/projected/25316150-6e7e-45ab-8d43-2bcf46ac51cb-kube-api-access-cp64v\") pod \"managed-serviceaccount-addon-agent-755bc595fd-j2wkm\" (UID: \"25316150-6e7e-45ab-8d43-2bcf46ac51cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm" Sep 10 23:37:29.456201 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.456202 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.456514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.456220 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25316150-6e7e-45ab-8d43-2bcf46ac51cb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-755bc595fd-j2wkm\" (UID: \"25316150-6e7e-45ab-8d43-2bcf46ac51cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm" Sep 10 23:37:29.456514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.456248 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-hub\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.456514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.456277 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.456514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.456305 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjsj\" (UniqueName: \"kubernetes.io/projected/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-kube-api-access-lkjsj\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.456514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.456329 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.456514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.456369 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-ca\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.457050 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.457000 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.458894 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.458868 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-hub\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.459027 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.458979 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.459105 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.459063 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.459373 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.459354 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-ca\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.459430 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.459357 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25316150-6e7e-45ab-8d43-2bcf46ac51cb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-755bc595fd-j2wkm\" (UID: \"25316150-6e7e-45ab-8d43-2bcf46ac51cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm" Sep 10 23:37:29.463818 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.463788 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp64v\" (UniqueName: \"kubernetes.io/projected/25316150-6e7e-45ab-8d43-2bcf46ac51cb-kube-api-access-cp64v\") pod \"managed-serviceaccount-addon-agent-755bc595fd-j2wkm\" (UID: \"25316150-6e7e-45ab-8d43-2bcf46ac51cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm" Sep 10 23:37:29.464415 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.464396 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjsj\" (UniqueName: \"kubernetes.io/projected/e62ba8d9-9dcf-4348-9f4d-65b62f9639ff-kube-api-access-lkjsj\") pod \"cluster-proxy-proxy-agent-86d6d6c7c4-cx98q\" (UID: \"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.607159 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.607137 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm" Sep 10 23:37:29.610257 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.610236 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" Sep 10 23:37:29.779047 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.779013 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7s59l" event={"ID":"3c94d0ce-50fc-4170-b7ad-87648ed95d4f","Type":"ContainerStarted","Data":"c4de2708d57564e9ac13aace9f95372e291ddba2602f2413b3baba5679755336"} Sep 10 23:37:29.799766 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.799706 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7s59l" podStartSLOduration=17.752839552 podStartE2EDuration="19.79969063s" totalImagesPullingTime="2.046851078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:37:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:29.798724886 +0000 UTC m=+57.101202898" watchObservedRunningTime="2026-09-10 23:37:29.79969063 +0000 UTC m=+57.102168636" Sep 10 23:37:29.881980 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.881857 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm"] Sep 10 23:37:29.884468 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:37:29.884446 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25316150_6e7e_45ab_8d43_2bcf46ac51cb.slice/crio-a49119addcc426fa920665cd7d94fcc3971b3e3e977e33abd683095a6b05a77f WatchSource:0}: Error finding container a49119addcc426fa920665cd7d94fcc3971b3e3e977e33abd683095a6b05a77f: Status 404 returned error can't find the container with id a49119addcc426fa920665cd7d94fcc3971b3e3e977e33abd683095a6b05a77f Sep 10 23:37:29.887170 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:29.887150 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q"] Sep 10 23:37:29.889274 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:37:29.889248 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode62ba8d9_9dcf_4348_9f4d_65b62f9639ff.slice/crio-6b13c8f0bdbaf4c1cc45a251367496932fb010e74fecf0d8a27f433caf964a33 WatchSource:0}: Error finding container 6b13c8f0bdbaf4c1cc45a251367496932fb010e74fecf0d8a27f433caf964a33: Status 404 returned error can't find the container with id 6b13c8f0bdbaf4c1cc45a251367496932fb010e74fecf0d8a27f433caf964a33 Sep 10 23:37:30.784107 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:30.784055 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm" event={"ID":"25316150-6e7e-45ab-8d43-2bcf46ac51cb","Type":"ContainerStarted","Data":"a49119addcc426fa920665cd7d94fcc3971b3e3e977e33abd683095a6b05a77f"} Sep 10 23:37:30.788313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:30.786320 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" event={"ID":"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff","Type":"ContainerStarted","Data":"6b13c8f0bdbaf4c1cc45a251367496932fb010e74fecf0d8a27f433caf964a33"} Sep 10 23:37:32.687756 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:32.687721 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rc9rz" Sep 10 23:37:33.799302 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:33.799271 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" event={"ID":"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff","Type":"ContainerStarted","Data":"55db01900e33de673e82ac1f1111d992a37bbf0274bcc8b6a361aa6dd438f9c6"} Sep 10 23:37:33.800640 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:33.800619 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm" event={"ID":"25316150-6e7e-45ab-8d43-2bcf46ac51cb","Type":"ContainerStarted","Data":"66d817380128c718d09a6388234bcfb0721bd16cc8e437a5eb518c032ec1d029"} Sep 10 23:37:33.815579 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:33.815523 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-755bc595fd-j2wkm" podStartSLOduration=0.980550163 podStartE2EDuration="4.815505418s" totalImagesPullingTime="3.834955255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:33.815388347 +0000 UTC m=+61.117866360" watchObservedRunningTime="2026-09-10 23:37:33.815505418 +0000 UTC m=+61.117983432" Sep 10 23:37:34.805894 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:34.805857 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" event={"ID":"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff","Type":"ContainerStarted","Data":"1bc8bee08724a97b5aa67a42f4e40a357e33e6151908b7991c35cee029129f2a"} Sep 10 23:37:34.805894 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:34.805897 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" event={"ID":"e62ba8d9-9dcf-4348-9f4d-65b62f9639ff","Type":"ContainerStarted","Data":"94319e593d849f92da0b0f7df152925a4edd5c9c6037560efad21b036109b895"} Sep 10 23:37:34.825156 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:34.825110 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86d6d6c7c4-cx98q" podStartSLOduration=1.979877737 podStartE2EDuration="5.825093107s" totalImagesPullingTime="3.84521537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:34.82371882 +0000 UTC m=+62.126196835" watchObservedRunningTime="2026-09-10 23:37:34.825093107 +0000 UTC m=+62.127571111" Sep 10 23:37:36.770962 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:36.770934 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lfxmq" Sep 10 23:37:38.932401 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:38.932358 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs\") pod \"network-metrics-daemon-7hv8v\" (UID: \"e030b330-e967-4317-bcd2-4664364ee362\") " pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:37:38.936963 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:38.936942 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 23:37:38.944809 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:38.944781 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e030b330-e967-4317-bcd2-4664364ee362-metrics-certs\") pod \"network-metrics-daemon-7hv8v\" (UID: \"e030b330-e967-4317-bcd2-4664364ee362\") " pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:37:39.033210 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.033176 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4zc\" (UniqueName: \"kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc\") pod \"network-check-target-hxfnv\" (UID: \"bf403608-4d7b-4585-88b9-7b1590654d3e\") " pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:37:39.033368 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.033223 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:37:39.036122 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.036099 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 23:37:39.036217 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.036201 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 23:37:39.046222 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.046199 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b40d527-4121-4aeb-8abd-c4c7c5f53688-original-pull-secret\") pod \"global-pull-secret-syncer-rt86f\" (UID: \"9b40d527-4121-4aeb-8abd-c4c7c5f53688\") " pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:37:39.046332 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.046239 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 23:37:39.057177 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.057158 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk4zc\" (UniqueName: \"kubernetes.io/projected/bf403608-4d7b-4585-88b9-7b1590654d3e-kube-api-access-fk4zc\") pod \"network-check-target-hxfnv\" (UID: \"bf403608-4d7b-4585-88b9-7b1590654d3e\") " pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:37:39.120773 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.120749 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xmh4\"" Sep 10 23:37:39.126877 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.126851 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t4h6l\"" Sep 10 23:37:39.128548 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.128533 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hv8v" Sep 10 23:37:39.134845 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.134822 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:37:39.212733 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.212704 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt86f" Sep 10 23:37:39.435451 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.435419 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7hv8v"] Sep 10 23:37:39.438815 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:37:39.438790 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode030b330_e967_4317_bcd2_4664364ee362.slice/crio-5ca7de5cac3f8fa624a31bdca506e7a483b301fb8ed4ceae790d560829994537 WatchSource:0}: Error finding container 5ca7de5cac3f8fa624a31bdca506e7a483b301fb8ed4ceae790d560829994537: Status 404 returned error can't find the container with id 5ca7de5cac3f8fa624a31bdca506e7a483b301fb8ed4ceae790d560829994537 Sep 10 23:37:39.460310 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.460259 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hxfnv"] Sep 10 23:37:39.462213 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:37:39.462189 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf403608_4d7b_4585_88b9_7b1590654d3e.slice/crio-5c7c7e95934be40fff0988f3a1dcb8b8cf26aec03986773f92778a0a29f73352 WatchSource:0}: Error finding container 5c7c7e95934be40fff0988f3a1dcb8b8cf26aec03986773f92778a0a29f73352: Status 404 returned error can't find the container with id 5c7c7e95934be40fff0988f3a1dcb8b8cf26aec03986773f92778a0a29f73352 Sep 10 23:37:39.511929 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.511904 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rt86f"] Sep 10 23:37:39.515208 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:37:39.515187 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b40d527_4121_4aeb_8abd_c4c7c5f53688.slice/crio-5d84565cdfbcdd89ebad05cc961285f460cfccace32a717a5305fa24b3742ef3 WatchSource:0}: Error finding container 5d84565cdfbcdd89ebad05cc961285f460cfccace32a717a5305fa24b3742ef3: Status 404 returned error can't find the container with id 5d84565cdfbcdd89ebad05cc961285f460cfccace32a717a5305fa24b3742ef3 Sep 10 23:37:39.822344 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.822251 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hxfnv" event={"ID":"bf403608-4d7b-4585-88b9-7b1590654d3e","Type":"ContainerStarted","Data":"5c7c7e95934be40fff0988f3a1dcb8b8cf26aec03986773f92778a0a29f73352"} Sep 10 23:37:39.823435 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.823393 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rt86f" event={"ID":"9b40d527-4121-4aeb-8abd-c4c7c5f53688","Type":"ContainerStarted","Data":"5d84565cdfbcdd89ebad05cc961285f460cfccace32a717a5305fa24b3742ef3"} Sep 10 23:37:39.824552 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:39.824529 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7hv8v" event={"ID":"e030b330-e967-4317-bcd2-4664364ee362","Type":"ContainerStarted","Data":"5ca7de5cac3f8fa624a31bdca506e7a483b301fb8ed4ceae790d560829994537"} Sep 10 23:37:41.836414 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:41.836373 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7hv8v" event={"ID":"e030b330-e967-4317-bcd2-4664364ee362","Type":"ContainerStarted","Data":"a429590cf0c3900f57c2f8e5f839b16b15133e49aac44190f941718c83e00e36"} Sep 10 23:37:41.836854 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:41.836439 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7hv8v" event={"ID":"e030b330-e967-4317-bcd2-4664364ee362","Type":"ContainerStarted","Data":"00d82f261af8cdedaacae478d9b153d22780c96bec53c947cbeeb7e23b8df678"} Sep 10 23:37:41.853353 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:41.853156 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7hv8v" podStartSLOduration=66.403827299 podStartE2EDuration="1m7.853137645s" totalImagesPullingTime="1.449310346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:41.851892357 +0000 UTC m=+69.154370372" watchObservedRunningTime="2026-09-10 23:37:41.853137645 +0000 UTC m=+69.155615661" Sep 10 23:37:44.849043 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:44.849005 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hxfnv" event={"ID":"bf403608-4d7b-4585-88b9-7b1590654d3e","Type":"ContainerStarted","Data":"3f4721c8fd0ae5d89c4ada99023fc57710331673f8550d57862ac9d42d42ea1a"} Sep 10 23:37:44.849494 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:44.849120 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:37:44.850590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:44.850568 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rt86f" event={"ID":"9b40d527-4121-4aeb-8abd-c4c7c5f53688","Type":"ContainerStarted","Data":"fe6b465bc5cedb00733c579ad50ede0b8c661304ae7b854aed28bb45e2918456"} Sep 10 23:37:44.865589 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:44.865541 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hxfnv" podStartSLOduration=67.244373556 podStartE2EDuration="1m11.865526564s" totalImagesPullingTime="4.621153008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:44.864619428 +0000 UTC m=+72.167097443" watchObservedRunningTime="2026-09-10 23:37:44.865526564 +0000 UTC m=+72.168004577" Sep 10 23:37:44.887875 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:44.887830 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rt86f" podStartSLOduration=66.31586604 podStartE2EDuration="1m10.887816936s" totalImagesPullingTime="4.571950896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:44.88748588 +0000 UTC m=+72.189963886" watchObservedRunningTime="2026-09-10 23:37:44.887816936 +0000 UTC m=+72.190294979" Sep 10 23:37:47.042281 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.042246 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bg2xd"] Sep 10 23:37:47.045468 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.045450 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.048089 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.048051 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 23:37:47.048220 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.048064 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 23:37:47.048383 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.048367 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 23:37:47.049154 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.049131 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 23:37:47.049264 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.049160 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 23:37:47.049264 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.049175 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 23:37:47.049264 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.049131 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gv69c\"" Sep 10 23:37:47.086816 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.086782 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-accelerators-collector-config\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.086958 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.086823 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39c93f55-cbf1-4a65-8386-dbe543395530-metrics-client-ca\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.086958 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.086858 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.086958 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.086916 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-textfile\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.087130 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.086966 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-wtmp\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.087130 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.087007 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-tls\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.087130 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.087024 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/39c93f55-cbf1-4a65-8386-dbe543395530-root\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.087130 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.087049 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv88n\" (UniqueName: \"kubernetes.io/projected/39c93f55-cbf1-4a65-8386-dbe543395530-kube-api-access-pv88n\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.087130 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.087117 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39c93f55-cbf1-4a65-8386-dbe543395530-sys\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.187465 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.187431 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-accelerators-collector-config\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.187621 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.187479 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39c93f55-cbf1-4a65-8386-dbe543395530-metrics-client-ca\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.187621 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.187507 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.187621 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.187538 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-textfile\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.187621 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.187564 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-wtmp\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.187621 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.187603 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-tls\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.187886 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.187648 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/39c93f55-cbf1-4a65-8386-dbe543395530-root\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.187886 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.187665 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv88n\" (UniqueName: \"kubernetes.io/projected/39c93f55-cbf1-4a65-8386-dbe543395530-kube-api-access-pv88n\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.187886 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.187694 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39c93f55-cbf1-4a65-8386-dbe543395530-sys\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.187886 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.187758 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39c93f55-cbf1-4a65-8386-dbe543395530-sys\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.187886 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.187766 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-wtmp\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.187886 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.187837 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/39c93f55-cbf1-4a65-8386-dbe543395530-root\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.188194 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.187981 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-textfile\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.188252 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.188232 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-accelerators-collector-config\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.188421 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.188402 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39c93f55-cbf1-4a65-8386-dbe543395530-metrics-client-ca\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.189864 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.189842 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.189935 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.189898 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/39c93f55-cbf1-4a65-8386-dbe543395530-node-exporter-tls\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.202809 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.202784 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv88n\" (UniqueName: \"kubernetes.io/projected/39c93f55-cbf1-4a65-8386-dbe543395530-kube-api-access-pv88n\") pod \"node-exporter-bg2xd\" (UID: \"39c93f55-cbf1-4a65-8386-dbe543395530\") " pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.355291 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.355263 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bg2xd" Sep 10 23:37:47.368243 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:37:47.368217 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c93f55_cbf1_4a65_8386_dbe543395530.slice/crio-1e4df96cc0788072f277f9a85856b9ff3569a50a42bda2529a228fcf9ff44af2 WatchSource:0}: Error finding container 1e4df96cc0788072f277f9a85856b9ff3569a50a42bda2529a228fcf9ff44af2: Status 404 returned error can't find the container with id 1e4df96cc0788072f277f9a85856b9ff3569a50a42bda2529a228fcf9ff44af2 Sep 10 23:37:47.861753 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:47.861718 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bg2xd" event={"ID":"39c93f55-cbf1-4a65-8386-dbe543395530","Type":"ContainerStarted","Data":"1e4df96cc0788072f277f9a85856b9ff3569a50a42bda2529a228fcf9ff44af2"} Sep 10 23:37:48.868547 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:48.868508 2583 generic.go:356] "Generic (PLEG): container finished" podID="39c93f55-cbf1-4a65-8386-dbe543395530" containerID="e9c72039519165bd8019540f99a52251ca83de813803402970469a9439cf47e5" exitCode=0 Sep 10 23:37:48.868907 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:48.868579 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bg2xd" event={"ID":"39c93f55-cbf1-4a65-8386-dbe543395530","Type":"ContainerDied","Data":"e9c72039519165bd8019540f99a52251ca83de813803402970469a9439cf47e5"} Sep 10 23:37:49.874312 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:49.874277 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bg2xd" event={"ID":"39c93f55-cbf1-4a65-8386-dbe543395530","Type":"ContainerStarted","Data":"11115255e5e215f7d5869e00f8857745c86417bb45ca655002fafaa44ab8528f"} Sep 10 23:37:49.874702 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:49.874329 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bg2xd" event={"ID":"39c93f55-cbf1-4a65-8386-dbe543395530","Type":"ContainerStarted","Data":"1f95b71de16cf4e32feb5b42fd3e54b1264ff2fd8beb1a0a0a2031cf5630736e"} Sep 10 23:37:49.899440 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:37:49.899393 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bg2xd" podStartSLOduration=1.788875274 podStartE2EDuration="2.899377573s" totalImagesPullingTime="1.048911344s" totalInitContainerRuntime="61.590955ms" isStatefulPod=true podCreationTimestamp="2026-09-10 23:37:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:49.898863116 +0000 UTC m=+77.201341125" watchObservedRunningTime="2026-09-10 23:37:49.899377573 +0000 UTC m=+77.201855584" Sep 10 23:38:15.856768 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:38:15.856734 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hxfnv" Sep 10 23:41:33.118160 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:33.118117 2583 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 23:41:54.021203 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.021172 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7"] Sep 10 23:41:54.024448 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.024433 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" Sep 10 23:41:54.028255 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.028237 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 23:41:54.028462 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.028447 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n9p52\"" Sep 10 23:41:54.029242 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.029221 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 23:41:54.036698 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.036674 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7"] Sep 10 23:41:54.144094 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.144032 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9bf82ca-66b7-434d-85af-f9308a4529bc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7\" (UID: \"b9bf82ca-66b7-434d-85af-f9308a4529bc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" Sep 10 23:41:54.144270 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.144110 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9bf82ca-66b7-434d-85af-f9308a4529bc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7\" (UID: \"b9bf82ca-66b7-434d-85af-f9308a4529bc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" Sep 10 23:41:54.144270 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.144177 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l5cv\" (UniqueName: \"kubernetes.io/projected/b9bf82ca-66b7-434d-85af-f9308a4529bc-kube-api-access-4l5cv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7\" (UID: \"b9bf82ca-66b7-434d-85af-f9308a4529bc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" Sep 10 23:41:54.245130 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.245059 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l5cv\" (UniqueName: \"kubernetes.io/projected/b9bf82ca-66b7-434d-85af-f9308a4529bc-kube-api-access-4l5cv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7\" (UID: \"b9bf82ca-66b7-434d-85af-f9308a4529bc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" Sep 10 23:41:54.245130 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.245132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9bf82ca-66b7-434d-85af-f9308a4529bc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7\" (UID: \"b9bf82ca-66b7-434d-85af-f9308a4529bc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" Sep 10 23:41:54.245394 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.245161 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9bf82ca-66b7-434d-85af-f9308a4529bc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7\" (UID: \"b9bf82ca-66b7-434d-85af-f9308a4529bc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" Sep 10 23:41:54.245533 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.245514 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9bf82ca-66b7-434d-85af-f9308a4529bc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7\" (UID: \"b9bf82ca-66b7-434d-85af-f9308a4529bc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" Sep 10 23:41:54.245603 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.245564 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9bf82ca-66b7-434d-85af-f9308a4529bc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7\" (UID: \"b9bf82ca-66b7-434d-85af-f9308a4529bc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" Sep 10 23:41:54.277107 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.277011 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l5cv\" (UniqueName: \"kubernetes.io/projected/b9bf82ca-66b7-434d-85af-f9308a4529bc-kube-api-access-4l5cv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7\" (UID: \"b9bf82ca-66b7-434d-85af-f9308a4529bc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" Sep 10 23:41:54.333052 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.333018 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" Sep 10 23:41:54.608621 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.608581 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7"] Sep 10 23:41:54.611864 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:41:54.611831 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bf82ca_66b7_434d_85af_f9308a4529bc.slice/crio-3a5facca7c258d56fe7821004e4310901062ea1f01dd5e83d0b09132be3c5c76 WatchSource:0}: Error finding container 3a5facca7c258d56fe7821004e4310901062ea1f01dd5e83d0b09132be3c5c76: Status 404 returned error can't find the container with id 3a5facca7c258d56fe7821004e4310901062ea1f01dd5e83d0b09132be3c5c76 Sep 10 23:41:54.613872 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.613856 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 23:41:54.664972 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.664938 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lfj79"] Sep 10 23:41:54.669043 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.669015 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" event={"ID":"b9bf82ca-66b7-434d-85af-f9308a4529bc","Type":"ContainerStarted","Data":"3a5facca7c258d56fe7821004e4310901062ea1f01dd5e83d0b09132be3c5c76"} Sep 10 23:41:54.669196 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.669146 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lfj79" Sep 10 23:41:54.671790 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.671761 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 23:41:54.671924 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.671795 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8tlhj\"" Sep 10 23:41:54.672055 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.672041 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 23:41:54.680957 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.680935 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lfj79"] Sep 10 23:41:54.850379 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.850346 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j7gb\" (UniqueName: \"kubernetes.io/projected/b10fba8e-752a-4326-a662-988e79e029dc-kube-api-access-7j7gb\") pod \"cert-manager-cainjector-559cdd8c87-lfj79\" (UID: \"b10fba8e-752a-4326-a662-988e79e029dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lfj79" Sep 10 23:41:54.850379 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.850383 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b10fba8e-752a-4326-a662-988e79e029dc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lfj79\" (UID: \"b10fba8e-752a-4326-a662-988e79e029dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lfj79" Sep 10 23:41:54.951328 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.951241 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j7gb\" (UniqueName: \"kubernetes.io/projected/b10fba8e-752a-4326-a662-988e79e029dc-kube-api-access-7j7gb\") pod \"cert-manager-cainjector-559cdd8c87-lfj79\" (UID: \"b10fba8e-752a-4326-a662-988e79e029dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lfj79" Sep 10 23:41:54.951328 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.951279 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b10fba8e-752a-4326-a662-988e79e029dc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lfj79\" (UID: \"b10fba8e-752a-4326-a662-988e79e029dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lfj79" Sep 10 23:41:54.962002 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.961975 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b10fba8e-752a-4326-a662-988e79e029dc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lfj79\" (UID: \"b10fba8e-752a-4326-a662-988e79e029dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lfj79" Sep 10 23:41:54.962002 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.962000 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j7gb\" (UniqueName: \"kubernetes.io/projected/b10fba8e-752a-4326-a662-988e79e029dc-kube-api-access-7j7gb\") pod \"cert-manager-cainjector-559cdd8c87-lfj79\" (UID: \"b10fba8e-752a-4326-a662-988e79e029dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lfj79" Sep 10 23:41:54.977826 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:54.977795 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lfj79" Sep 10 23:41:55.254061 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:55.253978 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lfj79"] Sep 10 23:41:55.257302 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:41:55.257273 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb10fba8e_752a_4326_a662_988e79e029dc.slice/crio-6a73e1934d4cadd4c84f3e51d32ca1e5aa6a778308b580647dd1c89a1509b456 WatchSource:0}: Error finding container 6a73e1934d4cadd4c84f3e51d32ca1e5aa6a778308b580647dd1c89a1509b456: Status 404 returned error can't find the container with id 6a73e1934d4cadd4c84f3e51d32ca1e5aa6a778308b580647dd1c89a1509b456 Sep 10 23:41:55.672044 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:55.672006 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lfj79" event={"ID":"b10fba8e-752a-4326-a662-988e79e029dc","Type":"ContainerStarted","Data":"6a73e1934d4cadd4c84f3e51d32ca1e5aa6a778308b580647dd1c89a1509b456"} Sep 10 23:41:58.687950 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:58.687910 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lfj79" event={"ID":"b10fba8e-752a-4326-a662-988e79e029dc","Type":"ContainerStarted","Data":"34eced46bcb5c0095387e31950d781c2c6a2743266f4e7c298f6bae05794aac0"} Sep 10 23:41:58.725217 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:41:58.725096 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lfj79" podStartSLOduration=1.514936361 podStartE2EDuration="4.725060886s" totalImagesPullingTime="3.210124525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:58.722822669 +0000 UTC m=+326.025300683" watchObservedRunningTime="2026-09-10 23:41:58.725060886 +0000 UTC m=+326.027538902" Sep 10 23:42:00.697293 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:00.697255 2583 generic.go:356] "Generic (PLEG): container finished" podID="b9bf82ca-66b7-434d-85af-f9308a4529bc" containerID="e44d31a2133d472d3fbabc160da10e38fd63753f00750d15a33e7eb64f756e0e" exitCode=0 Sep 10 23:42:00.697707 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:00.697332 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" event={"ID":"b9bf82ca-66b7-434d-85af-f9308a4529bc","Type":"ContainerDied","Data":"e44d31a2133d472d3fbabc160da10e38fd63753f00750d15a33e7eb64f756e0e"} Sep 10 23:42:02.706361 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:02.706331 2583 generic.go:356] "Generic (PLEG): container finished" podID="b9bf82ca-66b7-434d-85af-f9308a4529bc" containerID="ed3eb4fad07b28dbdb6427b0067b56bd854bab2eee8b6c7b054ca5c29436f3ab" exitCode=0 Sep 10 23:42:02.706744 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:02.706367 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" event={"ID":"b9bf82ca-66b7-434d-85af-f9308a4529bc","Type":"ContainerDied","Data":"ed3eb4fad07b28dbdb6427b0067b56bd854bab2eee8b6c7b054ca5c29436f3ab"} Sep 10 23:42:09.736235 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:09.736199 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" event={"ID":"b9bf82ca-66b7-434d-85af-f9308a4529bc","Type":"ContainerStarted","Data":"e3b491a14a8f0a75ed9a4aacabf5cdfd345c5e52562c68b838c71c268bcfa339"} Sep 10 23:42:09.761887 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:09.761835 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" podStartSLOduration=2.062394899 podStartE2EDuration="16.76182041s" totalImagesPullingTime="14.699425511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:09.761023911 +0000 UTC m=+337.063501925" watchObservedRunningTime="2026-09-10 23:42:09.76182041 +0000 UTC m=+337.064298422" Sep 10 23:42:10.741715 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:10.741667 2583 generic.go:356] "Generic (PLEG): container finished" podID="b9bf82ca-66b7-434d-85af-f9308a4529bc" containerID="e3b491a14a8f0a75ed9a4aacabf5cdfd345c5e52562c68b838c71c268bcfa339" exitCode=0 Sep 10 23:42:10.742065 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:10.741728 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" event={"ID":"b9bf82ca-66b7-434d-85af-f9308a4529bc","Type":"ContainerDied","Data":"e3b491a14a8f0a75ed9a4aacabf5cdfd345c5e52562c68b838c71c268bcfa339"} Sep 10 23:42:11.868058 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:11.868027 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" Sep 10 23:42:11.981021 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:11.980977 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9bf82ca-66b7-434d-85af-f9308a4529bc-bundle\") pod \"b9bf82ca-66b7-434d-85af-f9308a4529bc\" (UID: \"b9bf82ca-66b7-434d-85af-f9308a4529bc\") " Sep 10 23:42:11.981021 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:11.981030 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9bf82ca-66b7-434d-85af-f9308a4529bc-util\") pod \"b9bf82ca-66b7-434d-85af-f9308a4529bc\" (UID: \"b9bf82ca-66b7-434d-85af-f9308a4529bc\") " Sep 10 23:42:11.981260 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:11.981105 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l5cv\" (UniqueName: \"kubernetes.io/projected/b9bf82ca-66b7-434d-85af-f9308a4529bc-kube-api-access-4l5cv\") pod \"b9bf82ca-66b7-434d-85af-f9308a4529bc\" (UID: \"b9bf82ca-66b7-434d-85af-f9308a4529bc\") " Sep 10 23:42:11.981399 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:11.981373 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bf82ca-66b7-434d-85af-f9308a4529bc-bundle" (OuterVolumeSpecName: "bundle") pod "b9bf82ca-66b7-434d-85af-f9308a4529bc" (UID: "b9bf82ca-66b7-434d-85af-f9308a4529bc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:42:11.983377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:11.983351 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bf82ca-66b7-434d-85af-f9308a4529bc-kube-api-access-4l5cv" (OuterVolumeSpecName: "kube-api-access-4l5cv") pod "b9bf82ca-66b7-434d-85af-f9308a4529bc" (UID: "b9bf82ca-66b7-434d-85af-f9308a4529bc"). InnerVolumeSpecName "kube-api-access-4l5cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:42:11.985391 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:11.985358 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bf82ca-66b7-434d-85af-f9308a4529bc-util" (OuterVolumeSpecName: "util") pod "b9bf82ca-66b7-434d-85af-f9308a4529bc" (UID: "b9bf82ca-66b7-434d-85af-f9308a4529bc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:42:12.082111 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:12.081991 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l5cv\" (UniqueName: \"kubernetes.io/projected/b9bf82ca-66b7-434d-85af-f9308a4529bc-kube-api-access-4l5cv\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:42:12.082111 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:12.082025 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9bf82ca-66b7-434d-85af-f9308a4529bc-bundle\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:42:12.082111 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:12.082035 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9bf82ca-66b7-434d-85af-f9308a4529bc-util\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:42:12.750993 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:12.750954 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" event={"ID":"b9bf82ca-66b7-434d-85af-f9308a4529bc","Type":"ContainerDied","Data":"3a5facca7c258d56fe7821004e4310901062ea1f01dd5e83d0b09132be3c5c76"} Sep 10 23:42:12.751162 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:12.751002 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrlj7" Sep 10 23:42:12.751162 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:12.751005 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a5facca7c258d56fe7821004e4310901062ea1f01dd5e83d0b09132be3c5c76" Sep 10 23:42:17.339215 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.339183 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6"] Sep 10 23:42:17.339610 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.339407 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9bf82ca-66b7-434d-85af-f9308a4529bc" containerName="extract" Sep 10 23:42:17.339610 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.339417 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9bf82ca-66b7-434d-85af-f9308a4529bc" containerName="extract" Sep 10 23:42:17.339610 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.339431 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9bf82ca-66b7-434d-85af-f9308a4529bc" containerName="util" Sep 10 23:42:17.339610 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.339436 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9bf82ca-66b7-434d-85af-f9308a4529bc" containerName="util" Sep 10 23:42:17.339610 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.339448 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9bf82ca-66b7-434d-85af-f9308a4529bc" containerName="pull" Sep 10 23:42:17.339610 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.339453 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9bf82ca-66b7-434d-85af-f9308a4529bc" containerName="pull" Sep 10 23:42:17.339610 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.339487 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9bf82ca-66b7-434d-85af-f9308a4529bc" containerName="extract" Sep 10 23:42:17.397215 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.397183 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6"] Sep 10 23:42:17.397374 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.397287 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6" Sep 10 23:42:17.399844 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.399818 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 23:42:17.399980 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.399823 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hpvft\"" Sep 10 23:42:17.400692 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.400667 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 23:42:17.422258 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.422232 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c62f070a-0916-4aba-b33f-d32b9bf0916e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jdnr6\" (UID: \"c62f070a-0916-4aba-b33f-d32b9bf0916e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6" Sep 10 23:42:17.422380 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.422261 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbt26\" (UniqueName: \"kubernetes.io/projected/c62f070a-0916-4aba-b33f-d32b9bf0916e-kube-api-access-xbt26\") pod \"openshift-lws-operator-fd8ccff4c-jdnr6\" (UID: \"c62f070a-0916-4aba-b33f-d32b9bf0916e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6" Sep 10 23:42:17.522991 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.522949 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c62f070a-0916-4aba-b33f-d32b9bf0916e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jdnr6\" (UID: \"c62f070a-0916-4aba-b33f-d32b9bf0916e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6" Sep 10 23:42:17.522991 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.522992 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbt26\" (UniqueName: \"kubernetes.io/projected/c62f070a-0916-4aba-b33f-d32b9bf0916e-kube-api-access-xbt26\") pod \"openshift-lws-operator-fd8ccff4c-jdnr6\" (UID: \"c62f070a-0916-4aba-b33f-d32b9bf0916e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6" Sep 10 23:42:17.523382 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.523360 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c62f070a-0916-4aba-b33f-d32b9bf0916e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jdnr6\" (UID: \"c62f070a-0916-4aba-b33f-d32b9bf0916e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6" Sep 10 23:42:17.531691 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.531666 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbt26\" (UniqueName: \"kubernetes.io/projected/c62f070a-0916-4aba-b33f-d32b9bf0916e-kube-api-access-xbt26\") pod \"openshift-lws-operator-fd8ccff4c-jdnr6\" (UID: \"c62f070a-0916-4aba-b33f-d32b9bf0916e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6" Sep 10 23:42:17.706395 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.706359 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6" Sep 10 23:42:17.988415 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:17.988339 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6"] Sep 10 23:42:18.775303 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:18.775263 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6" event={"ID":"c62f070a-0916-4aba-b33f-d32b9bf0916e","Type":"ContainerStarted","Data":"3adbff3cd8871246edf53b48287ec42f88b7b6b766f923eede5c31bc6c87ea3a"} Sep 10 23:42:20.790308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:20.790269 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6" event={"ID":"c62f070a-0916-4aba-b33f-d32b9bf0916e","Type":"ContainerStarted","Data":"063790c04cd8e8a7763f2ac3ee00f913b3f4525506dafa2b1eec93fd49a84c5a"} Sep 10 23:42:20.806393 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:20.806344 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdnr6" podStartSLOduration=1.4973305080000001 podStartE2EDuration="3.806323023s" totalImagesPullingTime="2.308992515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:20.805608629 +0000 UTC m=+348.108086642" watchObservedRunningTime="2026-09-10 23:42:20.806323023 +0000 UTC m=+348.108801036" Sep 10 23:42:27.875981 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:27.875897 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl"] Sep 10 23:42:27.878759 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:27.878743 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:27.883912 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:27.883887 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 23:42:27.885046 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:27.885023 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tswhv\"" Sep 10 23:42:27.885139 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:27.885086 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 23:42:27.885139 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:27.885101 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 23:42:27.952781 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:27.952750 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl"] Sep 10 23:42:27.996860 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:27.996823 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95221866-eb45-44fc-bf0e-0b389769776a-cert\") pod \"lws-controller-manager-98d67b8cf-wswwl\" (UID: \"95221866-eb45-44fc-bf0e-0b389769776a\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:27.996860 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:27.996863 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/95221866-eb45-44fc-bf0e-0b389769776a-manager-config\") pod \"lws-controller-manager-98d67b8cf-wswwl\" (UID: \"95221866-eb45-44fc-bf0e-0b389769776a\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:27.997064 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:27.996883 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/95221866-eb45-44fc-bf0e-0b389769776a-metrics-cert\") pod \"lws-controller-manager-98d67b8cf-wswwl\" (UID: \"95221866-eb45-44fc-bf0e-0b389769776a\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:27.997064 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:27.996992 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7cjb\" (UniqueName: \"kubernetes.io/projected/95221866-eb45-44fc-bf0e-0b389769776a-kube-api-access-g7cjb\") pod \"lws-controller-manager-98d67b8cf-wswwl\" (UID: \"95221866-eb45-44fc-bf0e-0b389769776a\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:28.097619 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:28.097577 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7cjb\" (UniqueName: \"kubernetes.io/projected/95221866-eb45-44fc-bf0e-0b389769776a-kube-api-access-g7cjb\") pod \"lws-controller-manager-98d67b8cf-wswwl\" (UID: \"95221866-eb45-44fc-bf0e-0b389769776a\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:28.097799 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:28.097630 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95221866-eb45-44fc-bf0e-0b389769776a-cert\") pod \"lws-controller-manager-98d67b8cf-wswwl\" (UID: \"95221866-eb45-44fc-bf0e-0b389769776a\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:28.097799 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:28.097743 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/95221866-eb45-44fc-bf0e-0b389769776a-manager-config\") pod \"lws-controller-manager-98d67b8cf-wswwl\" (UID: \"95221866-eb45-44fc-bf0e-0b389769776a\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:28.097799 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:28.097788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/95221866-eb45-44fc-bf0e-0b389769776a-metrics-cert\") pod \"lws-controller-manager-98d67b8cf-wswwl\" (UID: \"95221866-eb45-44fc-bf0e-0b389769776a\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:28.098465 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:28.098445 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/95221866-eb45-44fc-bf0e-0b389769776a-manager-config\") pod \"lws-controller-manager-98d67b8cf-wswwl\" (UID: \"95221866-eb45-44fc-bf0e-0b389769776a\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:28.100213 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:28.100193 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95221866-eb45-44fc-bf0e-0b389769776a-cert\") pod \"lws-controller-manager-98d67b8cf-wswwl\" (UID: \"95221866-eb45-44fc-bf0e-0b389769776a\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:28.100394 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:28.100375 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/95221866-eb45-44fc-bf0e-0b389769776a-metrics-cert\") pod \"lws-controller-manager-98d67b8cf-wswwl\" (UID: \"95221866-eb45-44fc-bf0e-0b389769776a\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:28.114174 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:28.114140 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7cjb\" (UniqueName: \"kubernetes.io/projected/95221866-eb45-44fc-bf0e-0b389769776a-kube-api-access-g7cjb\") pod \"lws-controller-manager-98d67b8cf-wswwl\" (UID: \"95221866-eb45-44fc-bf0e-0b389769776a\") " pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:28.187096 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:28.186981 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:28.465106 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:28.465003 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl"] Sep 10 23:42:28.468361 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:42:28.468335 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95221866_eb45_44fc_bf0e_0b389769776a.slice/crio-c898d5571da6bcdf3bb3a4901d55ce0fd346298b903c4ba9ea4975e99a8a0060 WatchSource:0}: Error finding container c898d5571da6bcdf3bb3a4901d55ce0fd346298b903c4ba9ea4975e99a8a0060: Status 404 returned error can't find the container with id c898d5571da6bcdf3bb3a4901d55ce0fd346298b903c4ba9ea4975e99a8a0060 Sep 10 23:42:28.821784 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:28.821695 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" event={"ID":"95221866-eb45-44fc-bf0e-0b389769776a","Type":"ContainerStarted","Data":"c898d5571da6bcdf3bb3a4901d55ce0fd346298b903c4ba9ea4975e99a8a0060"} Sep 10 23:42:30.831995 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:30.831956 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" event={"ID":"95221866-eb45-44fc-bf0e-0b389769776a","Type":"ContainerStarted","Data":"a1b9fea47fbc56f472f1cd49166a595acaac6b0c3fb2a11c0a870b14453b521b"} Sep 10 23:42:30.832396 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:30.832080 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:30.867917 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:30.867863 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" podStartSLOduration=2.44553596 podStartE2EDuration="3.867845275s" totalImagesPullingTime="1.422309315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:30.866705538 +0000 UTC m=+358.169183551" watchObservedRunningTime="2026-09-10 23:42:30.867845275 +0000 UTC m=+358.170323286" Sep 10 23:42:38.891861 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:38.891824 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5"] Sep 10 23:42:38.895224 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:38.895207 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:38.899333 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:38.899314 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 23:42:38.899333 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:38.899323 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 23:42:38.899503 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:38.899485 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 23:42:38.899576 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:38.899558 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zkqzc\"" Sep 10 23:42:38.899692 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:38.899678 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 23:42:38.911478 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:38.911450 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5"] Sep 10 23:42:38.978514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:38.978481 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/04334b39-1b36-4759-b81d-877da7cf4e37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:38.978514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:38.978518 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/04334b39-1b36-4759-b81d-877da7cf4e37-manifests\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:38.978728 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:38.978596 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04334b39-1b36-4759-b81d-877da7cf4e37-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:38.978728 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:38.978641 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbcdv\" (UniqueName: \"kubernetes.io/projected/04334b39-1b36-4759-b81d-877da7cf4e37-kube-api-access-lbcdv\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:38.978728 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:38.978694 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04334b39-1b36-4759-b81d-877da7cf4e37-tmp\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:39.079191 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:39.079153 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04334b39-1b36-4759-b81d-877da7cf4e37-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:39.079191 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:39.079192 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbcdv\" (UniqueName: \"kubernetes.io/projected/04334b39-1b36-4759-b81d-877da7cf4e37-kube-api-access-lbcdv\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:39.079393 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:39.079329 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04334b39-1b36-4759-b81d-877da7cf4e37-tmp\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:39.079434 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:39.079403 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/04334b39-1b36-4759-b81d-877da7cf4e37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:39.079480 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:39.079443 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/04334b39-1b36-4759-b81d-877da7cf4e37-manifests\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:39.079828 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:39.079801 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/04334b39-1b36-4759-b81d-877da7cf4e37-manifests\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:39.081546 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:39.081522 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04334b39-1b36-4759-b81d-877da7cf4e37-tmp\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:39.081664 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:39.081646 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/04334b39-1b36-4759-b81d-877da7cf4e37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:39.081720 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:39.081698 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04334b39-1b36-4759-b81d-877da7cf4e37-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:39.086939 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:39.086912 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbcdv\" (UniqueName: \"kubernetes.io/projected/04334b39-1b36-4759-b81d-877da7cf4e37-kube-api-access-lbcdv\") pod \"opendatahub-operator-controller-manager-5bb6f54779-5mph5\" (UID: \"04334b39-1b36-4759-b81d-877da7cf4e37\") " pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:39.204903 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:39.204813 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:39.483726 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:39.483626 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5"] Sep 10 23:42:39.486904 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:42:39.486868 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04334b39_1b36_4759_b81d_877da7cf4e37.slice/crio-66088b9ab8dd44e14cb0305de0877a513e2a4083cb312d556ec35f41e56403bf WatchSource:0}: Error finding container 66088b9ab8dd44e14cb0305de0877a513e2a4083cb312d556ec35f41e56403bf: Status 404 returned error can't find the container with id 66088b9ab8dd44e14cb0305de0877a513e2a4083cb312d556ec35f41e56403bf Sep 10 23:42:39.864005 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:39.863972 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" event={"ID":"04334b39-1b36-4759-b81d-877da7cf4e37","Type":"ContainerStarted","Data":"66088b9ab8dd44e14cb0305de0877a513e2a4083cb312d556ec35f41e56403bf"} Sep 10 23:42:41.837998 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:41.837961 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-98d67b8cf-wswwl" Sep 10 23:42:42.879683 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:42.879647 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" event={"ID":"04334b39-1b36-4759-b81d-877da7cf4e37","Type":"ContainerStarted","Data":"2b3c32b844dad4f6983d4a990f55560e88b7b92c38efbaeb70239eddc903daaa"} Sep 10 23:42:43.884535 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:43.884501 2583 generic.go:356] "Generic (PLEG): container finished" podID="04334b39-1b36-4759-b81d-877da7cf4e37" containerID="2b3c32b844dad4f6983d4a990f55560e88b7b92c38efbaeb70239eddc903daaa" exitCode=0 Sep 10 23:42:43.884900 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:43.884596 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" event={"ID":"04334b39-1b36-4759-b81d-877da7cf4e37","Type":"ContainerDied","Data":"2b3c32b844dad4f6983d4a990f55560e88b7b92c38efbaeb70239eddc903daaa"} Sep 10 23:42:44.889968 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:44.889928 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" event={"ID":"04334b39-1b36-4759-b81d-877da7cf4e37","Type":"ContainerStarted","Data":"d3b82262bbaf5af943758c80f76119070ffd26f4924205e8fb545ff816aac1c3"} Sep 10 23:42:44.890373 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:44.890106 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:42:44.912793 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:44.912742 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" podStartSLOduration=2.582769558 podStartE2EDuration="6.912630382s" totalImagesPullingTime="3.303764582s" totalInitContainerRuntime="1.026096242s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:44.911538458 +0000 UTC m=+372.214016470" watchObservedRunningTime="2026-09-10 23:42:44.912630382 +0000 UTC m=+372.215108395" Sep 10 23:42:55.895968 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:42:55.895933 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5bb6f54779-5mph5" Sep 10 23:43:09.096180 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:09.096148 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-twk4s"] Sep 10 23:43:09.102266 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:09.102241 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" Sep 10 23:43:09.104600 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:09.104581 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 10 23:43:09.109391 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:09.109260 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-twk4s"] Sep 10 23:43:09.199949 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:09.199906 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a-tls-certs\") pod \"model-serving-api-54d89bd5fc-twk4s\" (UID: \"7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a\") " pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" Sep 10 23:43:09.200144 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:09.200016 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmkh9\" (UniqueName: \"kubernetes.io/projected/7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a-kube-api-access-vmkh9\") pod \"model-serving-api-54d89bd5fc-twk4s\" (UID: \"7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a\") " pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" Sep 10 23:43:09.301279 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:09.301229 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a-tls-certs\") pod \"model-serving-api-54d89bd5fc-twk4s\" (UID: \"7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a\") " pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" Sep 10 23:43:09.301477 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:09.301327 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmkh9\" (UniqueName: \"kubernetes.io/projected/7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a-kube-api-access-vmkh9\") pod \"model-serving-api-54d89bd5fc-twk4s\" (UID: \"7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a\") " pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" Sep 10 23:43:09.301477 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:43:09.301360 2583 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 10 23:43:09.301477 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:43:09.301468 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a-tls-certs podName:7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a nodeName:}" failed. No retries permitted until 2026-09-10 23:43:09.801447475 +0000 UTC m=+397.103925485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a-tls-certs") pod "model-serving-api-54d89bd5fc-twk4s" (UID: "7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a") : secret "model-serving-api-tls" not found Sep 10 23:43:09.310270 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:09.310238 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmkh9\" (UniqueName: \"kubernetes.io/projected/7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a-kube-api-access-vmkh9\") pod \"model-serving-api-54d89bd5fc-twk4s\" (UID: \"7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a\") " pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" Sep 10 23:43:09.805620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:09.805589 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a-tls-certs\") pod \"model-serving-api-54d89bd5fc-twk4s\" (UID: \"7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a\") " pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" Sep 10 23:43:09.807919 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:09.807894 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a-tls-certs\") pod \"model-serving-api-54d89bd5fc-twk4s\" (UID: \"7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a\") " pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" Sep 10 23:43:10.013321 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:10.013284 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" Sep 10 23:43:10.285920 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:10.285895 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-twk4s"] Sep 10 23:43:10.288848 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:43:10.288805 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e927681_ac8b_4b2e_b3f8_0fd5c3ac3b2a.slice/crio-28e456553473165b4298a214896d62281601bb7c343db3b065b74aeca591f347 WatchSource:0}: Error finding container 28e456553473165b4298a214896d62281601bb7c343db3b065b74aeca591f347: Status 404 returned error can't find the container with id 28e456553473165b4298a214896d62281601bb7c343db3b065b74aeca591f347 Sep 10 23:43:10.989569 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:10.989529 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" event={"ID":"7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a","Type":"ContainerStarted","Data":"28e456553473165b4298a214896d62281601bb7c343db3b065b74aeca591f347"} Sep 10 23:43:12.999814 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:12.999780 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" event={"ID":"7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a","Type":"ContainerStarted","Data":"d579ea58f0e29c35d930fd2c988cf78ac53d9b6b33565199c86caa3cd7f2872f"} Sep 10 23:43:13.000289 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:12.999883 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" Sep 10 23:43:13.018495 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:13.018452 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" podStartSLOduration=1.709567856 podStartE2EDuration="4.018436155s" totalImagesPullingTime="2.308868299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:43:13.016822469 +0000 UTC m=+400.319300483" watchObservedRunningTime="2026-09-10 23:43:13.018436155 +0000 UTC m=+400.320914185" Sep 10 23:43:14.581867 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.581832 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn"] Sep 10 23:43:14.587475 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.587454 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.589969 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.589943 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5mdzc\"" Sep 10 23:43:14.590101 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.589996 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 23:43:14.590172 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.590159 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 23:43:14.590774 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.590760 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 23:43:14.598008 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.597985 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn"] Sep 10 23:43:14.743808 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.743762 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.743808 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.743813 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc3a1d4-817a-4da5-918e-12c098448c9e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.744059 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.743890 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.744059 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.743940 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxf2c\" (UniqueName: \"kubernetes.io/projected/afc3a1d4-817a-4da5-918e-12c098448c9e-kube-api-access-pxf2c\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.744059 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.744019 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.744059 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.744047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc3a1d4-817a-4da5-918e-12c098448c9e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.744295 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.744097 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.744295 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.744160 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc3a1d4-817a-4da5-918e-12c098448c9e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.744295 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.744191 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.844822 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.844734 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.844822 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.844777 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxf2c\" (UniqueName: \"kubernetes.io/projected/afc3a1d4-817a-4da5-918e-12c098448c9e-kube-api-access-pxf2c\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.844822 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.844805 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.845112 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.844851 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc3a1d4-817a-4da5-918e-12c098448c9e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.845112 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.844896 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.845112 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.844920 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc3a1d4-817a-4da5-918e-12c098448c9e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.845112 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.844947 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.845112 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.844989 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.845112 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.845021 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc3a1d4-817a-4da5-918e-12c098448c9e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.845438 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.845350 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.845514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.845446 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.845514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.845456 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.845616 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.845600 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.845959 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.845942 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc3a1d4-817a-4da5-918e-12c098448c9e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.847362 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.847340 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc3a1d4-817a-4da5-918e-12c098448c9e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.847471 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.847393 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc3a1d4-817a-4da5-918e-12c098448c9e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.855513 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.855485 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxf2c\" (UniqueName: \"kubernetes.io/projected/afc3a1d4-817a-4da5-918e-12c098448c9e-kube-api-access-pxf2c\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.855670 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.855651 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc3a1d4-817a-4da5-918e-12c098448c9e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn\" (UID: \"afc3a1d4-817a-4da5-918e-12c098448c9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:14.900269 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:14.900235 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:15.190977 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:15.190945 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn"] Sep 10 23:43:15.193945 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:43:15.193911 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc3a1d4_817a_4da5_918e_12c098448c9e.slice/crio-90a20f14733b2563ed1029dfadd8da0b50c5a6c61710f9af0432e552e816eeff WatchSource:0}: Error finding container 90a20f14733b2563ed1029dfadd8da0b50c5a6c61710f9af0432e552e816eeff: Status 404 returned error can't find the container with id 90a20f14733b2563ed1029dfadd8da0b50c5a6c61710f9af0432e552e816eeff Sep 10 23:43:16.014812 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.014775 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" event={"ID":"afc3a1d4-817a-4da5-918e-12c098448c9e","Type":"ContainerStarted","Data":"90a20f14733b2563ed1029dfadd8da0b50c5a6c61710f9af0432e552e816eeff"} Sep 10 23:43:16.818629 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.818587 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs"] Sep 10 23:43:16.822060 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.822032 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:16.824854 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.824652 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 23:43:16.824854 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.824752 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4p88p\"" Sep 10 23:43:16.824854 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.824659 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 23:43:16.825883 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.825816 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 23:43:16.835037 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.835012 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs"] Sep 10 23:43:16.964889 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.964849 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v62mj\" (UniqueName: \"kubernetes.io/projected/73466eaa-d499-4209-be94-ba88149422b1-kube-api-access-v62mj\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:16.965111 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.964901 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/73466eaa-d499-4209-be94-ba88149422b1-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:16.965111 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.964995 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73466eaa-d499-4209-be94-ba88149422b1-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:16.965111 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.965021 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73466eaa-d499-4209-be94-ba88149422b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:16.965304 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.965183 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73466eaa-d499-4209-be94-ba88149422b1-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:16.965304 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.965228 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73466eaa-d499-4209-be94-ba88149422b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:16.965304 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:16.965256 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73466eaa-d499-4209-be94-ba88149422b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.066194 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.066154 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v62mj\" (UniqueName: \"kubernetes.io/projected/73466eaa-d499-4209-be94-ba88149422b1-kube-api-access-v62mj\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.066194 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.066190 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/73466eaa-d499-4209-be94-ba88149422b1-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.066715 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.066228 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73466eaa-d499-4209-be94-ba88149422b1-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.066715 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.066246 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73466eaa-d499-4209-be94-ba88149422b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.066715 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.066341 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73466eaa-d499-4209-be94-ba88149422b1-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.066715 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.066377 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73466eaa-d499-4209-be94-ba88149422b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.066715 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.066428 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73466eaa-d499-4209-be94-ba88149422b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.067338 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.067307 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73466eaa-d499-4209-be94-ba88149422b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.068878 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.068819 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73466eaa-d499-4209-be94-ba88149422b1-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.069109 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.069061 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73466eaa-d499-4209-be94-ba88149422b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.069213 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.069115 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/73466eaa-d499-4209-be94-ba88149422b1-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.069324 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.069300 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73466eaa-d499-4209-be94-ba88149422b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.077581 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.077535 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73466eaa-d499-4209-be94-ba88149422b1-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.077792 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.077760 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v62mj\" (UniqueName: \"kubernetes.io/projected/73466eaa-d499-4209-be94-ba88149422b1-kube-api-access-v62mj\") pod \"istiod-openshift-gateway-5677ccb49b-zcqfs\" (UID: \"73466eaa-d499-4209-be94-ba88149422b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.139426 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.139392 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:17.886628 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.886581 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:43:17.886749 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.886681 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:43:17.886749 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:17.886721 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:43:18.033255 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:18.033218 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" event={"ID":"afc3a1d4-817a-4da5-918e-12c098448c9e","Type":"ContainerStarted","Data":"be468b6fb32e9e9867994bbb7d82f4b85ac325ffdff9910abe4b316cc597b723"} Sep 10 23:43:18.078949 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:18.078851 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" podStartSLOduration=1.388533495 podStartE2EDuration="4.078833476s" totalImagesPullingTime="2.690299981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:43:18.071781727 +0000 UTC m=+405.374259739" watchObservedRunningTime="2026-09-10 23:43:18.078833476 +0000 UTC m=+405.381311552" Sep 10 23:43:18.138838 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:18.138772 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs"] Sep 10 23:43:18.141555 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:43:18.141523 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73466eaa_d499_4209_be94_ba88149422b1.slice/crio-6fbaf1530b4535cf9a3f4c9e8d16da4afe4c64355025690f2f34e9c4362dbd8e WatchSource:0}: Error finding container 6fbaf1530b4535cf9a3f4c9e8d16da4afe4c64355025690f2f34e9c4362dbd8e: Status 404 returned error can't find the container with id 6fbaf1530b4535cf9a3f4c9e8d16da4afe4c64355025690f2f34e9c4362dbd8e Sep 10 23:43:18.901285 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:18.901242 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:18.906984 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:18.906935 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:19.039493 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:19.039449 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" event={"ID":"73466eaa-d499-4209-be94-ba88149422b1","Type":"ContainerStarted","Data":"6fbaf1530b4535cf9a3f4c9e8d16da4afe4c64355025690f2f34e9c4362dbd8e"} Sep 10 23:43:19.039869 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:19.039843 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:19.040827 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:19.040806 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn" Sep 10 23:43:20.555842 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:20.555807 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:43:20.556183 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:20.555872 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:43:21.051676 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:21.051633 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" event={"ID":"73466eaa-d499-4209-be94-ba88149422b1","Type":"ContainerStarted","Data":"2675edbbc9c02c9afe78452b48e20cb2734eafdb26d28a78c14486411ab73a4d"} Sep 10 23:43:21.081358 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:21.081306 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" podStartSLOduration=2.669817025 podStartE2EDuration="5.081290435s" totalImagesPullingTime="2.41147341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:43:21.079497289 +0000 UTC m=+408.381975305" watchObservedRunningTime="2026-09-10 23:43:21.081290435 +0000 UTC m=+408.383768447" Sep 10 23:43:22.055691 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:22.055657 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:22.057319 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:22.057297 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zcqfs" Sep 10 23:43:24.008475 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:43:24.008446 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-twk4s" Sep 10 23:47:20.838366 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:20.838312 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9"] Sep 10 23:47:20.841537 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:20.841512 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:20.844104 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:20.844044 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ds5n5\"" Sep 10 23:47:20.844225 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:20.844106 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 10 23:47:20.844225 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:20.844178 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 23:47:20.844953 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:20.844936 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 23:47:20.851193 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:20.851171 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9"] Sep 10 23:47:20.992596 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:20.992563 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/088af8e1-9142-4c01-816c-25202e79cf1b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:20.992770 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:20.992619 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrjxj\" (UniqueName: \"kubernetes.io/projected/088af8e1-9142-4c01-816c-25202e79cf1b-kube-api-access-lrjxj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:20.992770 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:20.992680 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/088af8e1-9142-4c01-816c-25202e79cf1b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:20.992770 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:20.992710 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/088af8e1-9142-4c01-816c-25202e79cf1b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:20.992770 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:20.992742 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/088af8e1-9142-4c01-816c-25202e79cf1b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:20.992770 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:20.992767 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/088af8e1-9142-4c01-816c-25202e79cf1b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.093728 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.093641 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrjxj\" (UniqueName: \"kubernetes.io/projected/088af8e1-9142-4c01-816c-25202e79cf1b-kube-api-access-lrjxj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.093728 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.093685 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/088af8e1-9142-4c01-816c-25202e79cf1b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.093728 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.093707 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/088af8e1-9142-4c01-816c-25202e79cf1b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.093728 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.093729 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/088af8e1-9142-4c01-816c-25202e79cf1b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.094062 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.093747 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/088af8e1-9142-4c01-816c-25202e79cf1b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.094062 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.093777 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/088af8e1-9142-4c01-816c-25202e79cf1b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.094191 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.094062 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/088af8e1-9142-4c01-816c-25202e79cf1b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.094191 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.094160 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/088af8e1-9142-4c01-816c-25202e79cf1b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.094253 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.094228 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/088af8e1-9142-4c01-816c-25202e79cf1b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.096028 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.096007 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/088af8e1-9142-4c01-816c-25202e79cf1b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.096196 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.096178 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/088af8e1-9142-4c01-816c-25202e79cf1b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.102581 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.102557 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrjxj\" (UniqueName: \"kubernetes.io/projected/088af8e1-9142-4c01-816c-25202e79cf1b-kube-api-access-lrjxj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k9dv9\" (UID: \"088af8e1-9142-4c01-816c-25202e79cf1b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.153550 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.153526 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:21.421285 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.421253 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9"] Sep 10 23:47:21.423807 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:47:21.423778 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088af8e1_9142_4c01_816c_25202e79cf1b.slice/crio-b02a21a527548081bb55e378603ded69a17d7f32d8fff180f1394f54626ba9d7 WatchSource:0}: Error finding container b02a21a527548081bb55e378603ded69a17d7f32d8fff180f1394f54626ba9d7: Status 404 returned error can't find the container with id b02a21a527548081bb55e378603ded69a17d7f32d8fff180f1394f54626ba9d7 Sep 10 23:47:21.425042 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:21.425023 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 23:47:22.025760 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:22.025722 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" event={"ID":"088af8e1-9142-4c01-816c-25202e79cf1b","Type":"ContainerStarted","Data":"b02a21a527548081bb55e378603ded69a17d7f32d8fff180f1394f54626ba9d7"} Sep 10 23:47:23.031885 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:23.031850 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" event={"ID":"088af8e1-9142-4c01-816c-25202e79cf1b","Type":"ContainerStarted","Data":"e3c55537a29c46692f5ed531f03cc788ce25741acd2dd8d885019d8090654aca"} Sep 10 23:47:23.052588 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:23.052539 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" podStartSLOduration=2.091532763 podStartE2EDuration="3.052523543s" totalImagesPullingTime="960.99078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:47:23.051148678 +0000 UTC m=+650.353626695" watchObservedRunningTime="2026-09-10 23:47:23.052523543 +0000 UTC m=+650.355001553" Sep 10 23:47:31.153969 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:31.153930 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:31.154445 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:31.154005 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:31.166015 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:31.165990 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:32.078641 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:32.078606 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k9dv9" Sep 10 23:47:40.932448 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:40.932411 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp"] Sep 10 23:47:40.937204 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:40.937178 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:40.939786 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:40.939761 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 10 23:47:40.944341 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:40.944316 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp"] Sep 10 23:47:40.950521 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:40.950495 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06dac9d-4784-426d-9280-db614b0a52a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:40.950521 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:40.950526 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b06dac9d-4784-426d-9280-db614b0a52a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:40.950708 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:40.950553 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfn89\" (UniqueName: \"kubernetes.io/projected/b06dac9d-4784-426d-9280-db614b0a52a8-kube-api-access-hfn89\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:40.950708 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:40.950573 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b06dac9d-4784-426d-9280-db614b0a52a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:40.951129 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:40.950711 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b06dac9d-4784-426d-9280-db614b0a52a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:40.952030 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:40.951993 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b06dac9d-4784-426d-9280-db614b0a52a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.052789 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.052749 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b06dac9d-4784-426d-9280-db614b0a52a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.052789 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.052796 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b06dac9d-4784-426d-9280-db614b0a52a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.053043 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.052825 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06dac9d-4784-426d-9280-db614b0a52a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.053043 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.052948 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b06dac9d-4784-426d-9280-db614b0a52a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.053043 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.052982 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfn89\" (UniqueName: \"kubernetes.io/projected/b06dac9d-4784-426d-9280-db614b0a52a8-kube-api-access-hfn89\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.053043 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.053012 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b06dac9d-4784-426d-9280-db614b0a52a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.053284 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.053260 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b06dac9d-4784-426d-9280-db614b0a52a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.053329 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.053298 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b06dac9d-4784-426d-9280-db614b0a52a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.053442 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.053423 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b06dac9d-4784-426d-9280-db614b0a52a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.055098 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.055065 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b06dac9d-4784-426d-9280-db614b0a52a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.055499 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.055480 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b06dac9d-4784-426d-9280-db614b0a52a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.065225 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.065201 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfn89\" (UniqueName: \"kubernetes.io/projected/b06dac9d-4784-426d-9280-db614b0a52a8-kube-api-access-hfn89\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp\" (UID: \"b06dac9d-4784-426d-9280-db614b0a52a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.249749 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.249657 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:41.520689 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:41.520665 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp"] Sep 10 23:47:41.522705 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:47:41.522682 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06dac9d_4784_426d_9280_db614b0a52a8.slice/crio-5c9c150db609f8b051c464bbadae6e41a684e429559f1db0392832816dffa978 WatchSource:0}: Error finding container 5c9c150db609f8b051c464bbadae6e41a684e429559f1db0392832816dffa978: Status 404 returned error can't find the container with id 5c9c150db609f8b051c464bbadae6e41a684e429559f1db0392832816dffa978 Sep 10 23:47:42.108240 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:42.108198 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" event={"ID":"b06dac9d-4784-426d-9280-db614b0a52a8","Type":"ContainerStarted","Data":"b5dc1ee69801e8244557aaa4ea5579cba8e9d15c4208cbf9e23d62e4a8467307"} Sep 10 23:47:42.108593 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:42.108264 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" event={"ID":"b06dac9d-4784-426d-9280-db614b0a52a8","Type":"ContainerStarted","Data":"5c9c150db609f8b051c464bbadae6e41a684e429559f1db0392832816dffa978"} Sep 10 23:47:42.125047 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:42.124998 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" podStartSLOduration=1.927730841 podStartE2EDuration="2.124981864s" totalImagesPullingTime="197.251023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:47:42.123885372 +0000 UTC m=+669.426363397" watchObservedRunningTime="2026-09-10 23:47:42.124981864 +0000 UTC m=+669.427459876" Sep 10 23:47:51.250268 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:51.250230 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:51.250685 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:51.250290 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:51.262988 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:51.262960 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:52.158826 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:52.158789 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp" Sep 10 23:47:55.423280 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.423242 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb"] Sep 10 23:47:55.427000 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.426976 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.429791 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.429768 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 23:47:55.436559 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.436531 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb"] Sep 10 23:47:55.464758 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.464732 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.464898 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.464768 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.464898 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.464791 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.464898 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.464806 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ht9n\" (UniqueName: \"kubernetes.io/projected/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-kube-api-access-7ht9n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.465050 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.464943 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.465050 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.464984 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.565501 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.565462 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.565501 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.565504 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.565755 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.565538 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.565755 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.565569 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.565755 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.565598 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.565755 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.565620 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ht9n\" (UniqueName: \"kubernetes.io/projected/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-kube-api-access-7ht9n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.565967 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.565940 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.566028 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.566007 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.566094 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.566046 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.567779 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.567752 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.568024 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.568006 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.573020 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.572994 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ht9n\" (UniqueName: \"kubernetes.io/projected/9cfacd2e-bc36-44f3-87c7-2516c3fc33c0-kube-api-access-7ht9n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ns8zb\" (UID: \"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:55.737884 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:55.737809 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:47:56.008811 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:56.008758 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb"] Sep 10 23:47:56.010975 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:47:56.010946 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cfacd2e_bc36_44f3_87c7_2516c3fc33c0.slice/crio-ff79607a68beff5e3a197936ad06b2a70018a60088d4964be92874ca0ddb51f9 WatchSource:0}: Error finding container ff79607a68beff5e3a197936ad06b2a70018a60088d4964be92874ca0ddb51f9: Status 404 returned error can't find the container with id ff79607a68beff5e3a197936ad06b2a70018a60088d4964be92874ca0ddb51f9 Sep 10 23:47:56.163618 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:56.163584 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" event={"ID":"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0","Type":"ContainerStarted","Data":"ff79607a68beff5e3a197936ad06b2a70018a60088d4964be92874ca0ddb51f9"} Sep 10 23:47:57.174318 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:57.174280 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" event={"ID":"9cfacd2e-bc36-44f3-87c7-2516c3fc33c0","Type":"ContainerStarted","Data":"95dee81a49102644179f64a7393023ea6ec67da20502e344da50c66344ca5d8f"} Sep 10 23:47:57.192198 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:47:57.192154 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" podStartSLOduration=2.020288153 podStartE2EDuration="2.192139344s" totalImagesPullingTime="171.851191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:47:57.190556871 +0000 UTC m=+684.493034888" watchObservedRunningTime="2026-09-10 23:47:57.192139344 +0000 UTC m=+684.494617356" Sep 10 23:48:05.738430 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:48:05.738385 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:48:05.738951 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:48:05.738450 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:48:05.750868 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:48:05.750838 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:48:06.221517 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:48:06.221482 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ns8zb" Sep 10 23:49:56.783372 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.783298 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj"] Sep 10 23:49:56.786430 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.786412 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.789474 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.789452 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4c5699-openshift-default-dockercfg-zxcqt\"" Sep 10 23:49:56.810834 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.810808 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj"] Sep 10 23:49:56.863797 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.863763 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-istio-data\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.863797 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.863798 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90546c99-e542-4446-931d-be95d2dbd11c-istiod-ca-cert\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.864002 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.863830 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90546c99-e542-4446-931d-be95d2dbd11c-istio-podinfo\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.864002 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.863847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90546c99-e542-4446-931d-be95d2dbd11c-istio-token\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.864002 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.863931 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-credential-socket\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.864161 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.864006 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-workload-socket\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.864161 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.864031 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-istio-envoy\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.864161 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.864148 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld6mh\" (UniqueName: \"kubernetes.io/projected/90546c99-e542-4446-931d-be95d2dbd11c-kube-api-access-ld6mh\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.864295 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.864198 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-workload-certs\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.965350 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.965311 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90546c99-e542-4446-931d-be95d2dbd11c-istio-podinfo\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.965533 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.965357 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90546c99-e542-4446-931d-be95d2dbd11c-istio-token\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.965533 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.965394 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-credential-socket\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.965533 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.965455 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-workload-socket\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.965533 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.965480 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-istio-envoy\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.965757 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.965538 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6mh\" (UniqueName: \"kubernetes.io/projected/90546c99-e542-4446-931d-be95d2dbd11c-kube-api-access-ld6mh\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.965757 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.965576 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-workload-certs\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.965757 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.965604 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-istio-data\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.965757 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.965635 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90546c99-e542-4446-931d-be95d2dbd11c-istiod-ca-cert\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.965958 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.965825 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-credential-socket\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.965958 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.965841 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-workload-socket\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.966092 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.966058 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-workload-certs\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.966166 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.966127 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-istio-data\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.966397 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.966378 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90546c99-e542-4446-931d-be95d2dbd11c-istiod-ca-cert\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.967803 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.967770 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-istio-envoy\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.968011 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.967993 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90546c99-e542-4446-931d-be95d2dbd11c-istio-podinfo\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.976299 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.976279 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld6mh\" (UniqueName: \"kubernetes.io/projected/90546c99-e542-4446-931d-be95d2dbd11c-kube-api-access-ld6mh\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:56.978225 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:56.978207 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90546c99-e542-4446-931d-be95d2dbd11c-istio-token\") pod \"e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:57.097276 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:57.097193 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:57.380343 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:57.380318 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj"] Sep 10 23:49:57.382269 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:49:57.382244 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90546c99_e542_4446_931d_be95d2dbd11c.slice/crio-66fe843fbe4b573210042049ab8fb5d8f2b4caa15573ca5756de8b1452eb2654 WatchSource:0}: Error finding container 66fe843fbe4b573210042049ab8fb5d8f2b4caa15573ca5756de8b1452eb2654: Status 404 returned error can't find the container with id 66fe843fbe4b573210042049ab8fb5d8f2b4caa15573ca5756de8b1452eb2654 Sep 10 23:49:57.384189 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:57.384158 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:49:57.384240 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:57.384218 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:49:57.384277 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:57.384244 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:49:57.657919 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:57.657839 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" event={"ID":"90546c99-e542-4446-931d-be95d2dbd11c","Type":"ContainerStarted","Data":"4b0ba308079182f0e0eb956e04d5d8479ba135526b3b8dc0481804400886ccca"} Sep 10 23:49:57.657919 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:57.657892 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" event={"ID":"90546c99-e542-4446-931d-be95d2dbd11c","Type":"ContainerStarted","Data":"66fe843fbe4b573210042049ab8fb5d8f2b4caa15573ca5756de8b1452eb2654"} Sep 10 23:49:57.680285 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:57.680232 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" podStartSLOduration=1.6802148319999999 podStartE2EDuration="1.680214832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:57.678303219 +0000 UTC m=+804.980781232" watchObservedRunningTime="2026-09-10 23:49:57.680214832 +0000 UTC m=+804.982692845" Sep 10 23:49:58.098204 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:58.098120 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:58.102975 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:58.102949 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:58.662329 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:58.662298 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:49:58.663383 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:49:58.663366 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:50:04.683831 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.683790 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz"] Sep 10 23:50:04.687516 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.687492 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.690227 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.690204 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-b3de77-openshift-default-dockercfg-xfhlf\"" Sep 10 23:50:04.703291 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.703264 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz"] Sep 10 23:50:04.834080 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.834044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpzxc\" (UniqueName: \"kubernetes.io/projected/dbdd3267-3062-4b90-a272-1bfa4798984a-kube-api-access-kpzxc\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.834243 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.834102 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-workload-certs\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.834243 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.834137 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdd3267-3062-4b90-a272-1bfa4798984a-istiod-ca-cert\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.834243 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.834205 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-workload-socket\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.834342 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.834277 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-credential-socket\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.834342 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.834309 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-envoy\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.834407 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.834351 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-podinfo\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.834407 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.834379 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-token\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.834482 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.834408 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-data\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.934936 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.934866 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-podinfo\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.934936 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.934901 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-token\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.934936 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.934923 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-data\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.935225 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.934952 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpzxc\" (UniqueName: \"kubernetes.io/projected/dbdd3267-3062-4b90-a272-1bfa4798984a-kube-api-access-kpzxc\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.935225 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.934968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-workload-certs\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.935225 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.935002 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdd3267-3062-4b90-a272-1bfa4798984a-istiod-ca-cert\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.935225 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.935034 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-workload-socket\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.935225 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.935065 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-credential-socket\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.935225 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.935117 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-envoy\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.935664 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.935636 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-data\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.935777 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.935661 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-workload-socket\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.935777 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.935687 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-workload-certs\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.935879 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.935802 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-credential-socket\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.936182 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.936155 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdd3267-3062-4b90-a272-1bfa4798984a-istiod-ca-cert\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.937439 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.937413 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-envoy\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.937587 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.937568 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-podinfo\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.945194 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.945171 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-token\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.945507 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.945489 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpzxc\" (UniqueName: \"kubernetes.io/projected/dbdd3267-3062-4b90-a272-1bfa4798984a-kube-api-access-kpzxc\") pod \"e2e-autores-b3de77-openshift-default-6794588794-bqjsz\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:04.998989 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:04.998956 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:05.290986 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.290956 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz"] Sep 10 23:50:05.293376 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:50:05.293351 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdd3267_3062_4b90_a272_1bfa4798984a.slice/crio-2dd1cd38394c576cc22a8291bb575c13ffbc86f8a742f7507210d54f779272bc WatchSource:0}: Error finding container 2dd1cd38394c576cc22a8291bb575c13ffbc86f8a742f7507210d54f779272bc: Status 404 returned error can't find the container with id 2dd1cd38394c576cc22a8291bb575c13ffbc86f8a742f7507210d54f779272bc Sep 10 23:50:05.295477 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.295441 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:05.295554 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.295510 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:05.295554 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.295540 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:05.360808 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.360781 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g"] Sep 10 23:50:05.364086 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.364049 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.366992 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.366974 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6c506b66-openshift-default-dockercfg-st42w\"" Sep 10 23:50:05.384687 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.384662 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g"] Sep 10 23:50:05.541437 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.541337 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-workload-socket\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.541437 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.541386 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-credential-socket\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.541649 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.541436 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-token\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.541649 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.541474 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68157db-b2d6-4263-bd50-0fe91eda9580-istiod-ca-cert\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.541649 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.541558 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-workload-certs\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.541649 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.541577 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbl9k\" (UniqueName: \"kubernetes.io/projected/d68157db-b2d6-4263-bd50-0fe91eda9580-kube-api-access-dbl9k\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.541847 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.541656 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-data\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.541847 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.541672 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-podinfo\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.541847 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.541692 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-envoy\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.642980 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.642944 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-workload-socket\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.642980 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.642982 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-credential-socket\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.643308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.643003 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-token\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.643308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.643026 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68157db-b2d6-4263-bd50-0fe91eda9580-istiod-ca-cert\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.643308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.643103 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-workload-certs\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.643308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.643122 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbl9k\" (UniqueName: \"kubernetes.io/projected/d68157db-b2d6-4263-bd50-0fe91eda9580-kube-api-access-dbl9k\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.643308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.643190 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-data\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.643308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.643209 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-podinfo\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.643308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.643229 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-envoy\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.643716 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.643508 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-credential-socket\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.643716 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.643552 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-workload-certs\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.643843 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.643819 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-workload-socket\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.643903 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.643856 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-data\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.644425 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.644393 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68157db-b2d6-4263-bd50-0fe91eda9580-istiod-ca-cert\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.645680 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.645657 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-envoy\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.646130 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.646114 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-podinfo\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.662229 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.662203 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbl9k\" (UniqueName: \"kubernetes.io/projected/d68157db-b2d6-4263-bd50-0fe91eda9580-kube-api-access-dbl9k\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.662696 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.662681 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-token\") pod \"e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.674837 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.674822 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:05.694737 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.694706 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" event={"ID":"dbdd3267-3062-4b90-a272-1bfa4798984a","Type":"ContainerStarted","Data":"bf859e29274b1ef820f0fbd99fd42081eeedcc094d813bf352d823c0f82a784a"} Sep 10 23:50:05.695140 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.694767 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" event={"ID":"dbdd3267-3062-4b90-a272-1bfa4798984a","Type":"ContainerStarted","Data":"2dd1cd38394c576cc22a8291bb575c13ffbc86f8a742f7507210d54f779272bc"} Sep 10 23:50:05.723634 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.723565 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" podStartSLOduration=1.723544872 podStartE2EDuration="1.723544872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:05.722455903 +0000 UTC m=+813.024933914" watchObservedRunningTime="2026-09-10 23:50:05.723544872 +0000 UTC m=+813.026022884" Sep 10 23:50:05.959918 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.959883 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g"] Sep 10 23:50:05.962475 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:50:05.962451 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68157db_b2d6_4263_bd50_0fe91eda9580.slice/crio-fb356ff6703d9fd6ef4453d01431f92b298cdc4686a2c70c1a1c778bfb95a5f7 WatchSource:0}: Error finding container fb356ff6703d9fd6ef4453d01431f92b298cdc4686a2c70c1a1c778bfb95a5f7: Status 404 returned error can't find the container with id fb356ff6703d9fd6ef4453d01431f92b298cdc4686a2c70c1a1c778bfb95a5f7 Sep 10 23:50:05.964856 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.964828 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:05.964911 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.964888 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:05.964962 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.964914 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:05.999502 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:05.999477 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:06.705458 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:06.705415 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" event={"ID":"d68157db-b2d6-4263-bd50-0fe91eda9580","Type":"ContainerStarted","Data":"ae825c5d38612501d032aac6dcd7614501b012e67fb9049c2c2f5b698bb185d2"} Sep 10 23:50:06.705920 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:06.705522 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" event={"ID":"d68157db-b2d6-4263-bd50-0fe91eda9580","Type":"ContainerStarted","Data":"fb356ff6703d9fd6ef4453d01431f92b298cdc4686a2c70c1a1c778bfb95a5f7"} Sep 10 23:50:06.732600 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:06.732548 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" podStartSLOduration=1.732532521 podStartE2EDuration="1.732532521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:06.730246666 +0000 UTC m=+814.032724677" watchObservedRunningTime="2026-09-10 23:50:06.732532521 +0000 UTC m=+814.035010532" Sep 10 23:50:07.003854 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:07.003771 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:07.675993 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:07.675948 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:07.681216 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:07.681188 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:07.709170 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:07.709139 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:07.709566 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:07.709195 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:07.709917 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:07.709896 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:07.709986 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:07.709946 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:14.704251 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:14.704210 2583 patch_prober.go:28] interesting pod/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 23:50:14.704664 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:14.704302 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" podUID="90546c99-e542-4446-931d-be95d2dbd11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 23:50:15.058599 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.058511 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2"] Sep 10 23:50:15.062180 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.062161 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.065336 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.065311 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-948e64-openshift-default-dockercfg-swmnt\"" Sep 10 23:50:15.078545 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.078516 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2"] Sep 10 23:50:15.128749 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.128723 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wllw\" (UniqueName: \"kubernetes.io/projected/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-kube-api-access-6wllw\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.128910 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.128763 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-credential-socket\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.128910 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.128793 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-workload-certs\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.128910 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.128832 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istiod-ca-cert\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.128910 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.128892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-workload-socket\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.129133 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.128957 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-token\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.129133 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.128979 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-data\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.129133 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.128995 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-envoy\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.129133 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.129010 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-podinfo\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.229643 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.229610 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-token\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.229643 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.229645 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-data\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.229839 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.229665 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-envoy\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.229839 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.229699 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-podinfo\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.229839 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.229727 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wllw\" (UniqueName: \"kubernetes.io/projected/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-kube-api-access-6wllw\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.229839 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.229764 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-credential-socket\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.230032 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.229918 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-workload-certs\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.230032 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.229968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istiod-ca-cert\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.230032 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.230015 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-workload-socket\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.230437 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.230127 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-data\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.230437 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.230377 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-workload-certs\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.230437 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.230411 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-credential-socket\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.230610 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.230530 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-workload-socket\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.230768 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.230748 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istiod-ca-cert\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.232133 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.232113 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-envoy\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.232451 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.232417 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-podinfo\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.238060 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.238041 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-token\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.238155 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.238136 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wllw\" (UniqueName: \"kubernetes.io/projected/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-kube-api-access-6wllw\") pod \"e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.375291 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.375263 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:15.654871 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.654828 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2"] Sep 10 23:50:15.657160 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:50:15.657134 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aeb6d3c_8e7c_4928_adac_1dc6749fa329.slice/crio-5247b1fc070a3660c5583b4b821da6d07afeba43729ba02f9682b1c1798ed246 WatchSource:0}: Error finding container 5247b1fc070a3660c5583b4b821da6d07afeba43729ba02f9682b1c1798ed246: Status 404 returned error can't find the container with id 5247b1fc070a3660c5583b4b821da6d07afeba43729ba02f9682b1c1798ed246 Sep 10 23:50:15.659708 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.659673 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:15.659778 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.659741 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:15.659778 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.659770 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:15.749532 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.749378 2583 generic.go:356] "Generic (PLEG): container finished" podID="90546c99-e542-4446-931d-be95d2dbd11c" containerID="4b0ba308079182f0e0eb956e04d5d8479ba135526b3b8dc0481804400886ccca" exitCode=0 Sep 10 23:50:15.749532 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.749490 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" event={"ID":"90546c99-e542-4446-931d-be95d2dbd11c","Type":"ContainerDied","Data":"4b0ba308079182f0e0eb956e04d5d8479ba135526b3b8dc0481804400886ccca"} Sep 10 23:50:15.749978 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.749962 2583 scope.go:117] "RemoveContainer" containerID="4b0ba308079182f0e0eb956e04d5d8479ba135526b3b8dc0481804400886ccca" Sep 10 23:50:15.751302 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.751266 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:15.751417 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.751354 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:15.751417 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.751400 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:15.752216 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.752185 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" event={"ID":"4aeb6d3c-8e7c-4928-adac-1dc6749fa329","Type":"ContainerStarted","Data":"e44bdd3a75cd62689fb3ef973a10a817dbcbe55bb7f200691cc3e7451167ca4a"} Sep 10 23:50:15.752324 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.752236 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" event={"ID":"4aeb6d3c-8e7c-4928-adac-1dc6749fa329","Type":"ContainerStarted","Data":"5247b1fc070a3660c5583b4b821da6d07afeba43729ba02f9682b1c1798ed246"} Sep 10 23:50:15.805233 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:15.805185 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" podStartSLOduration=0.805169842 podStartE2EDuration="805.169842ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:15.802265188 +0000 UTC m=+823.104743199" watchObservedRunningTime="2026-09-10 23:50:15.805169842 +0000 UTC m=+823.107647851" Sep 10 23:50:16.375780 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:16.375738 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:16.380694 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:16.380664 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:16.773147 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:16.772491 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" event={"ID":"90546c99-e542-4446-931d-be95d2dbd11c","Type":"ContainerStarted","Data":"d750ac31d277136f1e2d923153356df389400ac60fd5c8b9e3b0f85044da392a"} Sep 10 23:50:16.773147 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:16.773054 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:16.774056 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:16.774031 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:17.098062 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:17.097971 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:50:17.098237 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:17.098215 2583 patch_prober.go:28] interesting pod/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Sep 10 23:50:17.098289 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:17.098260 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" podUID="90546c99-e542-4446-931d-be95d2dbd11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Sep 10 23:50:18.097971 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:18.097936 2583 patch_prober.go:28] interesting pod/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Sep 10 23:50:18.098358 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:18.097993 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" podUID="90546c99-e542-4446-931d-be95d2dbd11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Sep 10 23:50:19.097774 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:19.097742 2583 patch_prober.go:28] interesting pod/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Sep 10 23:50:19.097988 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:19.097967 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" podUID="90546c99-e542-4446-931d-be95d2dbd11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Sep 10 23:50:20.098560 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:20.098530 2583 patch_prober.go:28] interesting pod/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Sep 10 23:50:20.099020 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:20.098990 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" podUID="90546c99-e542-4446-931d-be95d2dbd11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Sep 10 23:50:21.098215 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:21.098175 2583 patch_prober.go:28] interesting pod/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Sep 10 23:50:21.098393 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:21.098240 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" podUID="90546c99-e542-4446-931d-be95d2dbd11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Sep 10 23:50:22.188203 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:22.188158 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:50:22.188668 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:22.188515 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:50:22.189569 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:22.189537 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:50:23.450288 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:23.450256 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj"] Sep 10 23:50:23.453939 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:23.453920 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj" Sep 10 23:50:23.456393 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:23.456376 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 10 23:50:23.456494 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:23.456382 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 23:50:23.467851 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:23.467812 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj"] Sep 10 23:50:23.522090 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:23.522023 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4458b38f-2549-4c7e-bc4a-84dd9b81ab7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c506b66-698tj\" (UID: \"4458b38f-2549-4c7e-bc4a-84dd9b81ab7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj" Sep 10 23:50:23.623789 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:23.623751 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4458b38f-2549-4c7e-bc4a-84dd9b81ab7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c506b66-698tj\" (UID: \"4458b38f-2549-4c7e-bc4a-84dd9b81ab7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj" Sep 10 23:50:23.624441 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:23.624418 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4458b38f-2549-4c7e-bc4a-84dd9b81ab7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6c506b66-698tj\" (UID: \"4458b38f-2549-4c7e-bc4a-84dd9b81ab7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj" Sep 10 23:50:23.750261 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:23.750180 2583 patch_prober.go:28] interesting pod/e2e-autores-b3de77-openshift-default-6794588794-bqjsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 23:50:23.750408 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:23.750252 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 23:50:23.765411 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:23.765385 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj" Sep 10 23:50:24.045170 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:24.042278 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj"] Sep 10 23:50:24.047487 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:50:24.047453 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4458b38f_2549_4c7e_bc4a_84dd9b81ab7a.slice/crio-439f065811679beb621864975a594ba71988fec1c3a0cf8a44386fd49d0dd402 WatchSource:0}: Error finding container 439f065811679beb621864975a594ba71988fec1c3a0cf8a44386fd49d0dd402: Status 404 returned error can't find the container with id 439f065811679beb621864975a594ba71988fec1c3a0cf8a44386fd49d0dd402 Sep 10 23:50:24.817300 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:24.817198 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj" event={"ID":"4458b38f-2549-4c7e-bc4a-84dd9b81ab7a","Type":"ContainerStarted","Data":"439f065811679beb621864975a594ba71988fec1c3a0cf8a44386fd49d0dd402"} Sep 10 23:50:24.818991 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:24.818952 2583 generic.go:356] "Generic (PLEG): container finished" podID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerID="bf859e29274b1ef820f0fbd99fd42081eeedcc094d813bf352d823c0f82a784a" exitCode=0 Sep 10 23:50:24.819150 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:24.819042 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" event={"ID":"dbdd3267-3062-4b90-a272-1bfa4798984a","Type":"ContainerDied","Data":"bf859e29274b1ef820f0fbd99fd42081eeedcc094d813bf352d823c0f82a784a"} Sep 10 23:50:24.819590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:24.819566 2583 scope.go:117] "RemoveContainer" containerID="bf859e29274b1ef820f0fbd99fd42081eeedcc094d813bf352d823c0f82a784a" Sep 10 23:50:24.820998 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:24.820959 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:24.821105 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:24.821089 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:24.821173 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:24.821139 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:25.864959 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:25.864916 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" event={"ID":"dbdd3267-3062-4b90-a272-1bfa4798984a","Type":"ContainerStarted","Data":"78ce22a029a1d2266ce6dc29a121c29ece0084fc60b1f2a1ab8a454bc280f5a9"} Sep 10 23:50:26.000008 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:25.999974 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:26.000529 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:26.000492 2583 patch_prober.go:28] interesting pod/e2e-autores-b3de77-openshift-default-6794588794-bqjsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Sep 10 23:50:26.000581 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:26.000562 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Sep 10 23:50:26.999402 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:26.999363 2583 patch_prober.go:28] interesting pod/e2e-autores-b3de77-openshift-default-6794588794-bqjsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Sep 10 23:50:26.999888 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:26.999421 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Sep 10 23:50:27.877515 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:27.877476 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj" event={"ID":"4458b38f-2549-4c7e-bc4a-84dd9b81ab7a","Type":"ContainerStarted","Data":"8d44d5683bf4c8e8e366751ad670447cfcbc6a971a55cb355e673e179268756b"} Sep 10 23:50:28.000246 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:28.000154 2583 patch_prober.go:28] interesting pod/e2e-autores-b3de77-openshift-default-6794588794-bqjsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Sep 10 23:50:28.000246 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:28.000215 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Sep 10 23:50:28.884811 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:28.884777 2583 generic.go:356] "Generic (PLEG): container finished" podID="4458b38f-2549-4c7e-bc4a-84dd9b81ab7a" containerID="8d44d5683bf4c8e8e366751ad670447cfcbc6a971a55cb355e673e179268756b" exitCode=0 Sep 10 23:50:28.885110 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:28.884851 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj" event={"ID":"4458b38f-2549-4c7e-bc4a-84dd9b81ab7a","Type":"ContainerDied","Data":"8d44d5683bf4c8e8e366751ad670447cfcbc6a971a55cb355e673e179268756b"} Sep 10 23:50:28.999882 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:28.999846 2583 patch_prober.go:28] interesting pod/e2e-autores-b3de77-openshift-default-6794588794-bqjsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Sep 10 23:50:29.000052 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:28.999906 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Sep 10 23:50:29.999902 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:29.999860 2583 patch_prober.go:28] interesting pod/e2e-autores-b3de77-openshift-default-6794588794-bqjsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Sep 10 23:50:30.000274 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:29.999937 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Sep 10 23:50:30.068638 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:30.068608 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj" Sep 10 23:50:30.103221 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:30.103187 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4458b38f-2549-4c7e-bc4a-84dd9b81ab7a-maas-api-service-ca\") pod \"4458b38f-2549-4c7e-bc4a-84dd9b81ab7a\" (UID: \"4458b38f-2549-4c7e-bc4a-84dd9b81ab7a\") " Sep 10 23:50:30.103608 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:30.103576 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4458b38f-2549-4c7e-bc4a-84dd9b81ab7a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4458b38f-2549-4c7e-bc4a-84dd9b81ab7a" (UID: "4458b38f-2549-4c7e-bc4a-84dd9b81ab7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:50:30.204599 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:30.204523 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4458b38f-2549-4c7e-bc4a-84dd9b81ab7a-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:50:30.897478 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:30.897439 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj" event={"ID":"4458b38f-2549-4c7e-bc4a-84dd9b81ab7a","Type":"ContainerDied","Data":"439f065811679beb621864975a594ba71988fec1c3a0cf8a44386fd49d0dd402"} Sep 10 23:50:30.897478 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:30.897474 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj" Sep 10 23:50:30.897733 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:30.897493 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="439f065811679beb621864975a594ba71988fec1c3a0cf8a44386fd49d0dd402" Sep 10 23:50:30.999515 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:30.999477 2583 patch_prober.go:28] interesting pod/e2e-autores-b3de77-openshift-default-6794588794-bqjsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Sep 10 23:50:30.999671 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:30.999533 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Sep 10 23:50:32.003474 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:32.003439 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:32.003852 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:32.003817 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:32.004463 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:32.004447 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:50:32.907972 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:32.907874 2583 generic.go:356] "Generic (PLEG): container finished" podID="d68157db-b2d6-4263-bd50-0fe91eda9580" containerID="ae825c5d38612501d032aac6dcd7614501b012e67fb9049c2c2f5b698bb185d2" exitCode=0 Sep 10 23:50:32.907972 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:32.907953 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" event={"ID":"d68157db-b2d6-4263-bd50-0fe91eda9580","Type":"ContainerDied","Data":"ae825c5d38612501d032aac6dcd7614501b012e67fb9049c2c2f5b698bb185d2"} Sep 10 23:50:32.908543 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:32.908524 2583 scope.go:117] "RemoveContainer" containerID="ae825c5d38612501d032aac6dcd7614501b012e67fb9049c2c2f5b698bb185d2" Sep 10 23:50:32.909474 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:32.909445 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:32.909562 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:32.909510 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:32.909562 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:32.909542 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:33.914747 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:33.914714 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" event={"ID":"d68157db-b2d6-4263-bd50-0fe91eda9580","Type":"ContainerStarted","Data":"e17ecd7019c85a4239503797aa3030bd4f1e3d6f891f3dd4764c2b18d14f0a0c"} Sep 10 23:50:34.675698 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:34.675654 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:34.676665 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:34.676636 2583 patch_prober.go:28] interesting pod/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Sep 10 23:50:34.676943 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:34.676914 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" podUID="d68157db-b2d6-4263-bd50-0fe91eda9580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Sep 10 23:50:35.676324 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:35.676280 2583 patch_prober.go:28] interesting pod/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Sep 10 23:50:35.676815 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:35.676355 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" podUID="d68157db-b2d6-4263-bd50-0fe91eda9580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Sep 10 23:50:36.675625 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:36.675590 2583 patch_prober.go:28] interesting pod/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Sep 10 23:50:36.675931 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:36.675878 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" podUID="d68157db-b2d6-4263-bd50-0fe91eda9580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Sep 10 23:50:36.881167 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:36.881133 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj"] Sep 10 23:50:36.884585 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:36.884557 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6c506b66-698tj"] Sep 10 23:50:37.027828 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:37.027704 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g"] Sep 10 23:50:37.027996 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:37.027970 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" podUID="d68157db-b2d6-4263-bd50-0fe91eda9580" containerName="istio-proxy" containerID="cri-o://e17ecd7019c85a4239503797aa3030bd4f1e3d6f891f3dd4764c2b18d14f0a0c" gracePeriod=30 Sep 10 23:50:37.307316 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:37.307223 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4458b38f-2549-4c7e-bc4a-84dd9b81ab7a" path="/var/lib/kubelet/pods/4458b38f-2549-4c7e-bc4a-84dd9b81ab7a/volumes" Sep 10 23:50:37.709916 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:37.709881 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:38.047197 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.045347 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx"] Sep 10 23:50:38.047197 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.045847 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4458b38f-2549-4c7e-bc4a-84dd9b81ab7a" containerName="revoke-keys" Sep 10 23:50:38.047197 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.045867 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4458b38f-2549-4c7e-bc4a-84dd9b81ab7a" containerName="revoke-keys" Sep 10 23:50:38.047197 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.045968 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4458b38f-2549-4c7e-bc4a-84dd9b81ab7a" containerName="revoke-keys" Sep 10 23:50:38.058339 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.058312 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.061111 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.060535 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx"] Sep 10 23:50:38.062399 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.062377 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-113befcd-gw-openshift-default-dockercfg-vbrbf\"" Sep 10 23:50:38.195194 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.195159 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-workload-certs\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.195194 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.195195 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-token\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.195399 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.195278 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298c69b0-a88c-46ca-82c7-b73daca112e2-istiod-ca-cert\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.195399 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.195362 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-podinfo\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.195465 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.195457 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-envoy\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.195518 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.195503 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-workload-socket\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.195557 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.195526 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-data\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.195637 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.195555 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-credential-socket\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.195637 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.195577 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgbw8\" (UniqueName: \"kubernetes.io/projected/298c69b0-a88c-46ca-82c7-b73daca112e2-kube-api-access-xgbw8\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.296707 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.296656 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-workload-socket\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.296707 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.296705 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-data\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.296938 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.296831 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-credential-socket\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.296938 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.296870 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgbw8\" (UniqueName: \"kubernetes.io/projected/298c69b0-a88c-46ca-82c7-b73daca112e2-kube-api-access-xgbw8\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.296938 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.296910 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-workload-certs\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.296938 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.296930 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-token\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.297168 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.296971 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298c69b0-a88c-46ca-82c7-b73daca112e2-istiod-ca-cert\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.297168 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.297013 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-podinfo\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.297168 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.297115 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-envoy\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.297168 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.297147 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-data\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.297412 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.297362 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-credential-socket\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.297534 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.297514 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-workload-certs\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.297688 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.297656 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-workload-socket\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.297969 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.297945 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298c69b0-a88c-46ca-82c7-b73daca112e2-istiod-ca-cert\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.300197 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.300158 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-envoy\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.300393 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.300371 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-podinfo\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.305041 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.305021 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-token\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.305851 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.305821 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgbw8\" (UniqueName: \"kubernetes.io/projected/298c69b0-a88c-46ca-82c7-b73daca112e2-kube-api-access-xgbw8\") pod \"e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.374739 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.374697 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:38.685743 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.685709 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx"] Sep 10 23:50:38.688499 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:50:38.688466 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298c69b0_a88c_46ca_82c7_b73daca112e2.slice/crio-7196e283b2cd981b3d06acee5c7818491a97ca06ac06e94c4f082258cfa968d1 WatchSource:0}: Error finding container 7196e283b2cd981b3d06acee5c7818491a97ca06ac06e94c4f082258cfa968d1: Status 404 returned error can't find the container with id 7196e283b2cd981b3d06acee5c7818491a97ca06ac06e94c4f082258cfa968d1 Sep 10 23:50:38.691396 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.691366 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:38.691485 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.691434 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:38.691485 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.691465 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:38.951263 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.951148 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" event={"ID":"298c69b0-a88c-46ca-82c7-b73daca112e2","Type":"ContainerStarted","Data":"bfcf44d2bcd47ece9c67612978a43c80d973a9119ccb7ab502a09a79e3b445d9"} Sep 10 23:50:38.951263 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.951229 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" event={"ID":"298c69b0-a88c-46ca-82c7-b73daca112e2","Type":"ContainerStarted","Data":"7196e283b2cd981b3d06acee5c7818491a97ca06ac06e94c4f082258cfa968d1"} Sep 10 23:50:38.974413 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:38.974352 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" podStartSLOduration=0.974334206 podStartE2EDuration="974.334206ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:38.971845666 +0000 UTC m=+846.274323678" watchObservedRunningTime="2026-09-10 23:50:38.974334206 +0000 UTC m=+846.276812216" Sep 10 23:50:39.375251 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:39.375213 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:39.380298 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:39.380272 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:39.956239 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:39.956210 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:39.957194 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:39.957174 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:50:40.963288 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:40.963256 2583 generic.go:356] "Generic (PLEG): container finished" podID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerID="e44bdd3a75cd62689fb3ef973a10a817dbcbe55bb7f200691cc3e7451167ca4a" exitCode=0 Sep 10 23:50:40.963798 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:40.963335 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" event={"ID":"4aeb6d3c-8e7c-4928-adac-1dc6749fa329","Type":"ContainerDied","Data":"e44bdd3a75cd62689fb3ef973a10a817dbcbe55bb7f200691cc3e7451167ca4a"} Sep 10 23:50:40.963798 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:40.963744 2583 scope.go:117] "RemoveContainer" containerID="e44bdd3a75cd62689fb3ef973a10a817dbcbe55bb7f200691cc3e7451167ca4a" Sep 10 23:50:40.964806 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:40.964783 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:40.964890 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:40.964875 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:40.964932 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:40.964925 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:41.970107 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:41.970059 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" event={"ID":"4aeb6d3c-8e7c-4928-adac-1dc6749fa329","Type":"ContainerStarted","Data":"050d91f16e1974e3f0401ecc02d596d18584c595b2a2837758c5b45abf72c583"} Sep 10 23:50:42.376249 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:42.376208 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:42.377347 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:42.377317 2583 patch_prober.go:28] interesting pod/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Sep 10 23:50:42.377488 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:42.377389 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" podUID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Sep 10 23:50:43.376308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:43.376272 2583 patch_prober.go:28] interesting pod/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Sep 10 23:50:43.376757 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:43.376328 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" podUID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Sep 10 23:50:44.376620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:44.376585 2583 patch_prober.go:28] interesting pod/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Sep 10 23:50:44.377001 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:44.376639 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" podUID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Sep 10 23:50:45.375932 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.375891 2583 patch_prober.go:28] interesting pod/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Sep 10 23:50:45.376141 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.375970 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" podUID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Sep 10 23:50:45.488114 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.488062 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:45.577368 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577282 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-envoy\") pod \"d68157db-b2d6-4263-bd50-0fe91eda9580\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " Sep 10 23:50:45.577368 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577329 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-workload-socket\") pod \"d68157db-b2d6-4263-bd50-0fe91eda9580\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " Sep 10 23:50:45.577368 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577348 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-workload-certs\") pod \"d68157db-b2d6-4263-bd50-0fe91eda9580\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " Sep 10 23:50:45.577631 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577379 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbl9k\" (UniqueName: \"kubernetes.io/projected/d68157db-b2d6-4263-bd50-0fe91eda9580-kube-api-access-dbl9k\") pod \"d68157db-b2d6-4263-bd50-0fe91eda9580\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " Sep 10 23:50:45.577631 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577421 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-token\") pod \"d68157db-b2d6-4263-bd50-0fe91eda9580\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " Sep 10 23:50:45.577631 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577499 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-data\") pod \"d68157db-b2d6-4263-bd50-0fe91eda9580\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " Sep 10 23:50:45.577631 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577535 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68157db-b2d6-4263-bd50-0fe91eda9580-istiod-ca-cert\") pod \"d68157db-b2d6-4263-bd50-0fe91eda9580\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " Sep 10 23:50:45.577631 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577559 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-podinfo\") pod \"d68157db-b2d6-4263-bd50-0fe91eda9580\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " Sep 10 23:50:45.577631 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577590 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d68157db-b2d6-4263-bd50-0fe91eda9580" (UID: "d68157db-b2d6-4263-bd50-0fe91eda9580"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:45.577631 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577621 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-credential-socket\") pod \"d68157db-b2d6-4263-bd50-0fe91eda9580\" (UID: \"d68157db-b2d6-4263-bd50-0fe91eda9580\") " Sep 10 23:50:45.577974 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577693 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d68157db-b2d6-4263-bd50-0fe91eda9580" (UID: "d68157db-b2d6-4263-bd50-0fe91eda9580"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:45.577974 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577750 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-data" (OuterVolumeSpecName: "istio-data") pod "d68157db-b2d6-4263-bd50-0fe91eda9580" (UID: "d68157db-b2d6-4263-bd50-0fe91eda9580"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:45.577974 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577914 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68157db-b2d6-4263-bd50-0fe91eda9580-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d68157db-b2d6-4263-bd50-0fe91eda9580" (UID: "d68157db-b2d6-4263-bd50-0fe91eda9580"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:50:45.577974 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577962 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:50:45.577974 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577966 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d68157db-b2d6-4263-bd50-0fe91eda9580" (UID: "d68157db-b2d6-4263-bd50-0fe91eda9580"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:45.578248 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577982 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:50:45.578248 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.577997 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:50:45.579649 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.579626 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-token" (OuterVolumeSpecName: "istio-token") pod "d68157db-b2d6-4263-bd50-0fe91eda9580" (UID: "d68157db-b2d6-4263-bd50-0fe91eda9580"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:50:45.579757 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.579665 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68157db-b2d6-4263-bd50-0fe91eda9580-kube-api-access-dbl9k" (OuterVolumeSpecName: "kube-api-access-dbl9k") pod "d68157db-b2d6-4263-bd50-0fe91eda9580" (UID: "d68157db-b2d6-4263-bd50-0fe91eda9580"). InnerVolumeSpecName "kube-api-access-dbl9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:50:45.579757 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.579745 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d68157db-b2d6-4263-bd50-0fe91eda9580" (UID: "d68157db-b2d6-4263-bd50-0fe91eda9580"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:45.579757 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.579751 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d68157db-b2d6-4263-bd50-0fe91eda9580" (UID: "d68157db-b2d6-4263-bd50-0fe91eda9580"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:50:45.678456 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.678418 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68157db-b2d6-4263-bd50-0fe91eda9580-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:50:45.678456 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.678446 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:50:45.678456 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.678455 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:50:45.678456 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.678465 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:50:45.678708 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.678474 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbl9k\" (UniqueName: \"kubernetes.io/projected/d68157db-b2d6-4263-bd50-0fe91eda9580-kube-api-access-dbl9k\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:50:45.678708 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.678482 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68157db-b2d6-4263-bd50-0fe91eda9580-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:50:45.991946 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.991911 2583 generic.go:356] "Generic (PLEG): container finished" podID="d68157db-b2d6-4263-bd50-0fe91eda9580" containerID="e17ecd7019c85a4239503797aa3030bd4f1e3d6f891f3dd4764c2b18d14f0a0c" exitCode=0 Sep 10 23:50:45.992136 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.991976 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" Sep 10 23:50:45.992136 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.992000 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" event={"ID":"d68157db-b2d6-4263-bd50-0fe91eda9580","Type":"ContainerDied","Data":"e17ecd7019c85a4239503797aa3030bd4f1e3d6f891f3dd4764c2b18d14f0a0c"} Sep 10 23:50:45.992136 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.992063 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g" event={"ID":"d68157db-b2d6-4263-bd50-0fe91eda9580","Type":"ContainerDied","Data":"fb356ff6703d9fd6ef4453d01431f92b298cdc4686a2c70c1a1c778bfb95a5f7"} Sep 10 23:50:45.992136 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:45.992118 2583 scope.go:117] "RemoveContainer" containerID="e17ecd7019c85a4239503797aa3030bd4f1e3d6f891f3dd4764c2b18d14f0a0c" Sep 10 23:50:46.001433 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:46.001411 2583 scope.go:117] "RemoveContainer" containerID="ae825c5d38612501d032aac6dcd7614501b012e67fb9049c2c2f5b698bb185d2" Sep 10 23:50:46.010747 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:46.010730 2583 scope.go:117] "RemoveContainer" containerID="e17ecd7019c85a4239503797aa3030bd4f1e3d6f891f3dd4764c2b18d14f0a0c" Sep 10 23:50:46.011023 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:50:46.011004 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e17ecd7019c85a4239503797aa3030bd4f1e3d6f891f3dd4764c2b18d14f0a0c\": container with ID starting with e17ecd7019c85a4239503797aa3030bd4f1e3d6f891f3dd4764c2b18d14f0a0c not found: ID does not exist" containerID="e17ecd7019c85a4239503797aa3030bd4f1e3d6f891f3dd4764c2b18d14f0a0c" Sep 10 23:50:46.011105 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:46.011032 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e17ecd7019c85a4239503797aa3030bd4f1e3d6f891f3dd4764c2b18d14f0a0c"} err="failed to get container status \"e17ecd7019c85a4239503797aa3030bd4f1e3d6f891f3dd4764c2b18d14f0a0c\": rpc error: code = NotFound desc = could not find container \"e17ecd7019c85a4239503797aa3030bd4f1e3d6f891f3dd4764c2b18d14f0a0c\": container with ID starting with e17ecd7019c85a4239503797aa3030bd4f1e3d6f891f3dd4764c2b18d14f0a0c not found: ID does not exist" Sep 10 23:50:46.011105 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:46.011062 2583 scope.go:117] "RemoveContainer" containerID="ae825c5d38612501d032aac6dcd7614501b012e67fb9049c2c2f5b698bb185d2" Sep 10 23:50:46.011433 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:50:46.011348 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae825c5d38612501d032aac6dcd7614501b012e67fb9049c2c2f5b698bb185d2\": container with ID starting with ae825c5d38612501d032aac6dcd7614501b012e67fb9049c2c2f5b698bb185d2 not found: ID does not exist" containerID="ae825c5d38612501d032aac6dcd7614501b012e67fb9049c2c2f5b698bb185d2" Sep 10 23:50:46.011433 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:46.011378 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae825c5d38612501d032aac6dcd7614501b012e67fb9049c2c2f5b698bb185d2"} err="failed to get container status \"ae825c5d38612501d032aac6dcd7614501b012e67fb9049c2c2f5b698bb185d2\": rpc error: code = NotFound desc = could not find container \"ae825c5d38612501d032aac6dcd7614501b012e67fb9049c2c2f5b698bb185d2\": container with ID starting with ae825c5d38612501d032aac6dcd7614501b012e67fb9049c2c2f5b698bb185d2 not found: ID does not exist" Sep 10 23:50:46.013554 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:46.013535 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g"] Sep 10 23:50:46.034091 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:46.033578 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6c506b66-openshift-default-6f99688f84-jmt8g"] Sep 10 23:50:46.376418 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:46.376381 2583 patch_prober.go:28] interesting pod/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Sep 10 23:50:46.376581 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:46.376454 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" podUID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Sep 10 23:50:46.773189 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:46.773095 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:47.308183 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:47.308148 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d68157db-b2d6-4263-bd50-0fe91eda9580" path="/var/lib/kubelet/pods/d68157db-b2d6-4263-bd50-0fe91eda9580/volumes" Sep 10 23:50:47.379751 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:47.379723 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:50:47.380739 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:50:47.380716 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:51:01.078573 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:01.078472 2583 generic.go:356] "Generic (PLEG): container finished" podID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerID="bfcf44d2bcd47ece9c67612978a43c80d973a9119ccb7ab502a09a79e3b445d9" exitCode=0 Sep 10 23:51:01.078573 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:01.078516 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" event={"ID":"298c69b0-a88c-46ca-82c7-b73daca112e2","Type":"ContainerDied","Data":"bfcf44d2bcd47ece9c67612978a43c80d973a9119ccb7ab502a09a79e3b445d9"} Sep 10 23:51:01.079184 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:01.079003 2583 scope.go:117] "RemoveContainer" containerID="bfcf44d2bcd47ece9c67612978a43c80d973a9119ccb7ab502a09a79e3b445d9" Sep 10 23:51:01.080132 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:01.080103 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:01.080205 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:01.080176 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:01.080242 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:01.080229 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:02.084709 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:02.084673 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" event={"ID":"298c69b0-a88c-46ca-82c7-b73daca112e2","Type":"ContainerStarted","Data":"849714c08736ab92a84446c7cf65501c70d7c5bfd067c0ae18d7df010cf30aeb"} Sep 10 23:51:02.375018 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:02.374984 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:51:02.375269 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:02.375246 2583 patch_prober.go:28] interesting pod/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 10 23:51:02.375319 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:02.375295 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 10 23:51:03.375886 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:03.375855 2583 patch_prober.go:28] interesting pod/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 10 23:51:03.376313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:03.375904 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 10 23:51:04.375870 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:04.375830 2583 patch_prober.go:28] interesting pod/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 10 23:51:04.376123 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:04.375896 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 10 23:51:05.375112 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.375052 2583 patch_prober.go:28] interesting pod/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 10 23:51:05.375289 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.375133 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 10 23:51:05.410224 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.410191 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp"] Sep 10 23:51:05.410587 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.410539 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68157db-b2d6-4263-bd50-0fe91eda9580" containerName="istio-proxy" Sep 10 23:51:05.410587 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.410551 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68157db-b2d6-4263-bd50-0fe91eda9580" containerName="istio-proxy" Sep 10 23:51:05.410587 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.410563 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68157db-b2d6-4263-bd50-0fe91eda9580" containerName="istio-proxy" Sep 10 23:51:05.410587 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.410568 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68157db-b2d6-4263-bd50-0fe91eda9580" containerName="istio-proxy" Sep 10 23:51:05.410726 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.410615 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d68157db-b2d6-4263-bd50-0fe91eda9580" containerName="istio-proxy" Sep 10 23:51:05.410726 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.410624 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d68157db-b2d6-4263-bd50-0fe91eda9580" containerName="istio-proxy" Sep 10 23:51:05.413569 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.413555 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp" Sep 10 23:51:05.415910 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.415889 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 23:51:05.415910 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.415896 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 10 23:51:05.422204 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.422177 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp"] Sep 10 23:51:05.443591 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.443554 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2d0680-ebd6-4f73-afe1-b5e22167d3b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp\" (UID: \"8e2d0680-ebd6-4f73-afe1-b5e22167d3b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp" Sep 10 23:51:05.545018 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.544981 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2d0680-ebd6-4f73-afe1-b5e22167d3b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp\" (UID: \"8e2d0680-ebd6-4f73-afe1-b5e22167d3b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp" Sep 10 23:51:05.545861 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.545835 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2d0680-ebd6-4f73-afe1-b5e22167d3b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp\" (UID: \"8e2d0680-ebd6-4f73-afe1-b5e22167d3b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp" Sep 10 23:51:05.730174 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:05.730061 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp" Sep 10 23:51:06.005381 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:06.005358 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp"] Sep 10 23:51:06.007830 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:51:06.007803 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2d0680_ebd6_4f73_afe1_b5e22167d3b7.slice/crio-552646f68bfc15bbc039d677a5efc3d07660eb7a826711f4fbb37fea3027d9e2 WatchSource:0}: Error finding container 552646f68bfc15bbc039d677a5efc3d07660eb7a826711f4fbb37fea3027d9e2: Status 404 returned error can't find the container with id 552646f68bfc15bbc039d677a5efc3d07660eb7a826711f4fbb37fea3027d9e2 Sep 10 23:51:06.106046 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:06.106011 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp" event={"ID":"8e2d0680-ebd6-4f73-afe1-b5e22167d3b7","Type":"ContainerStarted","Data":"552646f68bfc15bbc039d677a5efc3d07660eb7a826711f4fbb37fea3027d9e2"} Sep 10 23:51:06.375261 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:06.375226 2583 patch_prober.go:28] interesting pod/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 10 23:51:06.375430 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:06.375296 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 10 23:51:07.112302 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:07.112266 2583 generic.go:356] "Generic (PLEG): container finished" podID="8e2d0680-ebd6-4f73-afe1-b5e22167d3b7" containerID="d020d276c9163c632a0dce2821559b0e69aca4d1556be75fe35c0234b1c99f05" exitCode=0 Sep 10 23:51:07.112661 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:07.112323 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp" event={"ID":"8e2d0680-ebd6-4f73-afe1-b5e22167d3b7","Type":"ContainerDied","Data":"d020d276c9163c632a0dce2821559b0e69aca4d1556be75fe35c0234b1c99f05"} Sep 10 23:51:07.375313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:07.375277 2583 patch_prober.go:28] interesting pod/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 10 23:51:07.375497 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:07.375329 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 10 23:51:08.242283 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:08.242258 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp" Sep 10 23:51:08.271389 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:08.271366 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2d0680-ebd6-4f73-afe1-b5e22167d3b7-maas-api-service-ca\") pod \"8e2d0680-ebd6-4f73-afe1-b5e22167d3b7\" (UID: \"8e2d0680-ebd6-4f73-afe1-b5e22167d3b7\") " Sep 10 23:51:08.271678 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:08.271653 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2d0680-ebd6-4f73-afe1-b5e22167d3b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e2d0680-ebd6-4f73-afe1-b5e22167d3b7" (UID: "8e2d0680-ebd6-4f73-afe1-b5e22167d3b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:51:08.372965 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:08.372856 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2d0680-ebd6-4f73-afe1-b5e22167d3b7-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:08.379549 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:08.379524 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:51:08.379776 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:08.379753 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:51:08.380411 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:08.380388 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:51:09.122346 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:09.122318 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp" Sep 10 23:51:09.122510 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:09.122356 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp" event={"ID":"8e2d0680-ebd6-4f73-afe1-b5e22167d3b7","Type":"ContainerDied","Data":"552646f68bfc15bbc039d677a5efc3d07660eb7a826711f4fbb37fea3027d9e2"} Sep 10 23:51:09.122510 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:09.122398 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="552646f68bfc15bbc039d677a5efc3d07660eb7a826711f4fbb37fea3027d9e2" Sep 10 23:51:12.607226 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:12.607190 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp"] Sep 10 23:51:12.610337 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:12.610309 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113befcd-w4hhp"] Sep 10 23:51:12.759082 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:12.759037 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx"] Sep 10 23:51:12.759287 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:12.759244 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerName="istio-proxy" containerID="cri-o://849714c08736ab92a84446c7cf65501c70d7c5bfd067c0ae18d7df010cf30aeb" gracePeriod=30 Sep 10 23:51:13.307926 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.307893 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e2d0680-ebd6-4f73-afe1-b5e22167d3b7" path="/var/lib/kubelet/pods/8e2d0680-ebd6-4f73-afe1-b5e22167d3b7/volumes" Sep 10 23:51:13.542756 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.542725 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb"] Sep 10 23:51:13.543132 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.543117 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e2d0680-ebd6-4f73-afe1-b5e22167d3b7" containerName="revoke-keys" Sep 10 23:51:13.543190 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.543134 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e2d0680-ebd6-4f73-afe1-b5e22167d3b7" containerName="revoke-keys" Sep 10 23:51:13.543231 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.543190 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e2d0680-ebd6-4f73-afe1-b5e22167d3b7" containerName="revoke-keys" Sep 10 23:51:13.546331 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.546314 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.549000 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.548975 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9566fcb0-openshift-default-dockercfg-8mnhj\"" Sep 10 23:51:13.560681 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.560624 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb"] Sep 10 23:51:13.620403 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.620365 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de6c23d-6018-4976-9d01-14773a965b30-istio-podinfo\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.620848 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.620422 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-istio-data\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.620848 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.620449 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de6c23d-6018-4976-9d01-14773a965b30-istio-token\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.620848 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.620478 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-credential-socket\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.620848 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.620504 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-workload-certs\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.620848 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.620551 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de6c23d-6018-4976-9d01-14773a965b30-istiod-ca-cert\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.620848 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.620593 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6pwh\" (UniqueName: \"kubernetes.io/projected/6de6c23d-6018-4976-9d01-14773a965b30-kube-api-access-r6pwh\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.620848 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.620636 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-workload-socket\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.620848 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.620714 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-istio-envoy\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.721652 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.721612 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de6c23d-6018-4976-9d01-14773a965b30-istio-podinfo\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.721844 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.721663 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-istio-data\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.721844 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.721689 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de6c23d-6018-4976-9d01-14773a965b30-istio-token\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.721844 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.721718 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-credential-socket\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.721844 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.721740 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-workload-certs\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.721844 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.721784 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de6c23d-6018-4976-9d01-14773a965b30-istiod-ca-cert\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.721844 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.721822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pwh\" (UniqueName: \"kubernetes.io/projected/6de6c23d-6018-4976-9d01-14773a965b30-kube-api-access-r6pwh\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.722185 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.721862 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-workload-socket\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.722185 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.721923 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-istio-envoy\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.722288 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.722208 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-istio-data\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.722288 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.722235 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-workload-certs\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.722391 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.722337 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-workload-socket\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.722391 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.722370 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-credential-socket\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.722594 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.722570 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de6c23d-6018-4976-9d01-14773a965b30-istiod-ca-cert\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.724117 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.724092 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-istio-envoy\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.724230 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.724101 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de6c23d-6018-4976-9d01-14773a965b30-istio-podinfo\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.730285 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.730262 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de6c23d-6018-4976-9d01-14773a965b30-istio-token\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.730824 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.730795 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6pwh\" (UniqueName: \"kubernetes.io/projected/6de6c23d-6018-4976-9d01-14773a965b30-kube-api-access-r6pwh\") pod \"e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:13.856286 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:13.856247 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:14.135308 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:14.135224 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb"] Sep 10 23:51:14.138841 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:51:14.138807 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de6c23d_6018_4976_9d01_14773a965b30.slice/crio-4840213dfa8e8059e8dba6307f7ade51a0010fbb878fa6399816be980051c79d WatchSource:0}: Error finding container 4840213dfa8e8059e8dba6307f7ade51a0010fbb878fa6399816be980051c79d: Status 404 returned error can't find the container with id 4840213dfa8e8059e8dba6307f7ade51a0010fbb878fa6399816be980051c79d Sep 10 23:51:14.141354 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:14.141322 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:14.141464 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:14.141397 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:14.141464 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:14.141437 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:14.145432 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:14.145408 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" event={"ID":"6de6c23d-6018-4976-9d01-14773a965b30","Type":"ContainerStarted","Data":"4840213dfa8e8059e8dba6307f7ade51a0010fbb878fa6399816be980051c79d"} Sep 10 23:51:15.151839 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:15.151795 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" event={"ID":"6de6c23d-6018-4976-9d01-14773a965b30","Type":"ContainerStarted","Data":"8d21f252559ae81f9d95e67c7771b7a322fa75e2dc5a30ef18c1407420e46d39"} Sep 10 23:51:15.175200 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:15.175139 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" podStartSLOduration=2.17512013 podStartE2EDuration="2.17512013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:15.174645754 +0000 UTC m=+882.477123767" watchObservedRunningTime="2026-09-10 23:51:15.17512013 +0000 UTC m=+882.477598144" Sep 10 23:51:15.857083 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:15.857033 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:15.862200 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:15.862175 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:16.157911 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:16.157823 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:16.158890 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:16.158869 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:18.004208 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.004184 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:51:18.064835 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.064762 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-workload-certs\") pod \"298c69b0-a88c-46ca-82c7-b73daca112e2\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " Sep 10 23:51:18.064835 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.064800 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-podinfo\") pod \"298c69b0-a88c-46ca-82c7-b73daca112e2\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " Sep 10 23:51:18.064835 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.064815 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-credential-socket\") pod \"298c69b0-a88c-46ca-82c7-b73daca112e2\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " Sep 10 23:51:18.064835 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.064834 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgbw8\" (UniqueName: \"kubernetes.io/projected/298c69b0-a88c-46ca-82c7-b73daca112e2-kube-api-access-xgbw8\") pod \"298c69b0-a88c-46ca-82c7-b73daca112e2\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " Sep 10 23:51:18.065141 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.064874 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-data\") pod \"298c69b0-a88c-46ca-82c7-b73daca112e2\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " Sep 10 23:51:18.065141 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.064911 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-token\") pod \"298c69b0-a88c-46ca-82c7-b73daca112e2\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " Sep 10 23:51:18.065141 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.064930 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-workload-socket\") pod \"298c69b0-a88c-46ca-82c7-b73daca112e2\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " Sep 10 23:51:18.065141 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.064956 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298c69b0-a88c-46ca-82c7-b73daca112e2-istiod-ca-cert\") pod \"298c69b0-a88c-46ca-82c7-b73daca112e2\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " Sep 10 23:51:18.065141 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.065042 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-envoy\") pod \"298c69b0-a88c-46ca-82c7-b73daca112e2\" (UID: \"298c69b0-a88c-46ca-82c7-b73daca112e2\") " Sep 10 23:51:18.065141 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.065045 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "298c69b0-a88c-46ca-82c7-b73daca112e2" (UID: "298c69b0-a88c-46ca-82c7-b73daca112e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:18.065458 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.065183 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "298c69b0-a88c-46ca-82c7-b73daca112e2" (UID: "298c69b0-a88c-46ca-82c7-b73daca112e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:18.065458 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.065252 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-data" (OuterVolumeSpecName: "istio-data") pod "298c69b0-a88c-46ca-82c7-b73daca112e2" (UID: "298c69b0-a88c-46ca-82c7-b73daca112e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:18.065458 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.065279 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "298c69b0-a88c-46ca-82c7-b73daca112e2" (UID: "298c69b0-a88c-46ca-82c7-b73daca112e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:18.065458 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.065408 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:18.065458 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.065428 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:18.065458 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.065441 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:18.065458 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.065454 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:18.065745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.065496 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298c69b0-a88c-46ca-82c7-b73daca112e2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "298c69b0-a88c-46ca-82c7-b73daca112e2" (UID: "298c69b0-a88c-46ca-82c7-b73daca112e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:51:18.067166 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.067137 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-token" (OuterVolumeSpecName: "istio-token") pod "298c69b0-a88c-46ca-82c7-b73daca112e2" (UID: "298c69b0-a88c-46ca-82c7-b73daca112e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:51:18.067273 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.067196 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "298c69b0-a88c-46ca-82c7-b73daca112e2" (UID: "298c69b0-a88c-46ca-82c7-b73daca112e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:18.067401 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.067372 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298c69b0-a88c-46ca-82c7-b73daca112e2-kube-api-access-xgbw8" (OuterVolumeSpecName: "kube-api-access-xgbw8") pod "298c69b0-a88c-46ca-82c7-b73daca112e2" (UID: "298c69b0-a88c-46ca-82c7-b73daca112e2"). InnerVolumeSpecName "kube-api-access-xgbw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:51:18.067401 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.067391 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "298c69b0-a88c-46ca-82c7-b73daca112e2" (UID: "298c69b0-a88c-46ca-82c7-b73daca112e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:51:18.166620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.166592 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:18.166620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.166618 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298c69b0-a88c-46ca-82c7-b73daca112e2-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:18.166787 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.166629 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:18.166787 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.166640 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298c69b0-a88c-46ca-82c7-b73daca112e2-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:18.166787 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.166653 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgbw8\" (UniqueName: \"kubernetes.io/projected/298c69b0-a88c-46ca-82c7-b73daca112e2-kube-api-access-xgbw8\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:18.169022 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.168993 2583 generic.go:356] "Generic (PLEG): container finished" podID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerID="849714c08736ab92a84446c7cf65501c70d7c5bfd067c0ae18d7df010cf30aeb" exitCode=0 Sep 10 23:51:18.169137 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.169049 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" Sep 10 23:51:18.169137 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.169086 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" event={"ID":"298c69b0-a88c-46ca-82c7-b73daca112e2","Type":"ContainerDied","Data":"849714c08736ab92a84446c7cf65501c70d7c5bfd067c0ae18d7df010cf30aeb"} Sep 10 23:51:18.169205 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.169138 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx" event={"ID":"298c69b0-a88c-46ca-82c7-b73daca112e2","Type":"ContainerDied","Data":"7196e283b2cd981b3d06acee5c7818491a97ca06ac06e94c4f082258cfa968d1"} Sep 10 23:51:18.169205 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.169160 2583 scope.go:117] "RemoveContainer" containerID="849714c08736ab92a84446c7cf65501c70d7c5bfd067c0ae18d7df010cf30aeb" Sep 10 23:51:18.178920 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.178904 2583 scope.go:117] "RemoveContainer" containerID="bfcf44d2bcd47ece9c67612978a43c80d973a9119ccb7ab502a09a79e3b445d9" Sep 10 23:51:18.186405 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.186390 2583 scope.go:117] "RemoveContainer" containerID="849714c08736ab92a84446c7cf65501c70d7c5bfd067c0ae18d7df010cf30aeb" Sep 10 23:51:18.186664 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:51:18.186645 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"849714c08736ab92a84446c7cf65501c70d7c5bfd067c0ae18d7df010cf30aeb\": container with ID starting with 849714c08736ab92a84446c7cf65501c70d7c5bfd067c0ae18d7df010cf30aeb not found: ID does not exist" containerID="849714c08736ab92a84446c7cf65501c70d7c5bfd067c0ae18d7df010cf30aeb" Sep 10 23:51:18.186740 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.186675 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849714c08736ab92a84446c7cf65501c70d7c5bfd067c0ae18d7df010cf30aeb"} err="failed to get container status \"849714c08736ab92a84446c7cf65501c70d7c5bfd067c0ae18d7df010cf30aeb\": rpc error: code = NotFound desc = could not find container \"849714c08736ab92a84446c7cf65501c70d7c5bfd067c0ae18d7df010cf30aeb\": container with ID starting with 849714c08736ab92a84446c7cf65501c70d7c5bfd067c0ae18d7df010cf30aeb not found: ID does not exist" Sep 10 23:51:18.186740 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.186698 2583 scope.go:117] "RemoveContainer" containerID="bfcf44d2bcd47ece9c67612978a43c80d973a9119ccb7ab502a09a79e3b445d9" Sep 10 23:51:18.186979 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:51:18.186962 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcf44d2bcd47ece9c67612978a43c80d973a9119ccb7ab502a09a79e3b445d9\": container with ID starting with bfcf44d2bcd47ece9c67612978a43c80d973a9119ccb7ab502a09a79e3b445d9 not found: ID does not exist" containerID="bfcf44d2bcd47ece9c67612978a43c80d973a9119ccb7ab502a09a79e3b445d9" Sep 10 23:51:18.187021 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.186985 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcf44d2bcd47ece9c67612978a43c80d973a9119ccb7ab502a09a79e3b445d9"} err="failed to get container status \"bfcf44d2bcd47ece9c67612978a43c80d973a9119ccb7ab502a09a79e3b445d9\": rpc error: code = NotFound desc = could not find container \"bfcf44d2bcd47ece9c67612978a43c80d973a9119ccb7ab502a09a79e3b445d9\": container with ID starting with bfcf44d2bcd47ece9c67612978a43c80d973a9119ccb7ab502a09a79e3b445d9 not found: ID does not exist" Sep 10 23:51:18.192977 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.192957 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx"] Sep 10 23:51:18.202835 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:18.202808 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-113befcd-gw-openshift-default-84545f5564-2pnfx"] Sep 10 23:51:19.313647 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:19.313608 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" path="/var/lib/kubelet/pods/298c69b0-a88c-46ca-82c7-b73daca112e2/volumes" Sep 10 23:51:39.276926 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:39.276892 2583 generic.go:356] "Generic (PLEG): container finished" podID="6de6c23d-6018-4976-9d01-14773a965b30" containerID="8d21f252559ae81f9d95e67c7771b7a322fa75e2dc5a30ef18c1407420e46d39" exitCode=0 Sep 10 23:51:39.277345 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:39.276964 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" event={"ID":"6de6c23d-6018-4976-9d01-14773a965b30","Type":"ContainerDied","Data":"8d21f252559ae81f9d95e67c7771b7a322fa75e2dc5a30ef18c1407420e46d39"} Sep 10 23:51:39.277386 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:39.277352 2583 scope.go:117] "RemoveContainer" containerID="8d21f252559ae81f9d95e67c7771b7a322fa75e2dc5a30ef18c1407420e46d39" Sep 10 23:51:39.278367 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:39.278342 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:39.278431 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:39.278408 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:39.278465 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:39.278439 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:40.283174 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:40.283134 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" event={"ID":"6de6c23d-6018-4976-9d01-14773a965b30","Type":"ContainerStarted","Data":"8de43562cc65949168f0ecf4c2a3175d1a32643725675cb1261c4de59b3ab67b"} Sep 10 23:51:40.856727 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:40.856681 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:40.856979 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:40.856954 2583 patch_prober.go:28] interesting pod/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 10 23:51:40.857062 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:40.857041 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" podUID="6de6c23d-6018-4976-9d01-14773a965b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 10 23:51:41.857147 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:41.857103 2583 patch_prober.go:28] interesting pod/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 10 23:51:41.857614 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:41.857167 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" podUID="6de6c23d-6018-4976-9d01-14773a965b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 10 23:51:42.175649 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.175572 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp"] Sep 10 23:51:42.175909 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.175898 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerName="istio-proxy" Sep 10 23:51:42.175952 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.175912 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerName="istio-proxy" Sep 10 23:51:42.175952 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.175924 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerName="istio-proxy" Sep 10 23:51:42.175952 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.175930 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerName="istio-proxy" Sep 10 23:51:42.176064 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.175989 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerName="istio-proxy" Sep 10 23:51:42.176064 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.175998 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="298c69b0-a88c-46ca-82c7-b73daca112e2" containerName="istio-proxy" Sep 10 23:51:42.178891 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.178872 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp" Sep 10 23:51:42.181063 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.181041 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 23:51:42.181184 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.181143 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 10 23:51:42.187145 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.187122 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp"] Sep 10 23:51:42.279455 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.279410 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9357a92b-c1e1-43bb-859a-aa64a9657b1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp\" (UID: \"9357a92b-c1e1-43bb-859a-aa64a9657b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp" Sep 10 23:51:42.380906 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.380868 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9357a92b-c1e1-43bb-859a-aa64a9657b1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp\" (UID: \"9357a92b-c1e1-43bb-859a-aa64a9657b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp" Sep 10 23:51:42.381626 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.381606 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9357a92b-c1e1-43bb-859a-aa64a9657b1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp\" (UID: \"9357a92b-c1e1-43bb-859a-aa64a9657b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp" Sep 10 23:51:42.490335 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.490247 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp" Sep 10 23:51:42.781494 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.781471 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp"] Sep 10 23:51:42.783563 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:51:42.783522 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9357a92b_c1e1_43bb_859a_aa64a9657b1b.slice/crio-e28a9ca6a3d2bf58fdb9e984bbaf15fd566986f5669fe1be5da9b4d17c1dc579 WatchSource:0}: Error finding container e28a9ca6a3d2bf58fdb9e984bbaf15fd566986f5669fe1be5da9b4d17c1dc579: Status 404 returned error can't find the container with id e28a9ca6a3d2bf58fdb9e984bbaf15fd566986f5669fe1be5da9b4d17c1dc579 Sep 10 23:51:42.856803 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.856771 2583 patch_prober.go:28] interesting pod/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 10 23:51:42.856962 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:42.856838 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" podUID="6de6c23d-6018-4976-9d01-14773a965b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 10 23:51:43.298643 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:43.298605 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp" event={"ID":"9357a92b-c1e1-43bb-859a-aa64a9657b1b","Type":"ContainerStarted","Data":"982134d65473497d6f79248f31ddf5da9d3df5224d59938765ad8d94671f0f5f"} Sep 10 23:51:43.299020 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:43.298673 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp" event={"ID":"9357a92b-c1e1-43bb-859a-aa64a9657b1b","Type":"ContainerStarted","Data":"e28a9ca6a3d2bf58fdb9e984bbaf15fd566986f5669fe1be5da9b4d17c1dc579"} Sep 10 23:51:43.349234 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:43.349184 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp" podStartSLOduration=1.34916822 podStartE2EDuration="1.34916822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:43.344971164 +0000 UTC m=+910.647449176" watchObservedRunningTime="2026-09-10 23:51:43.34916822 +0000 UTC m=+910.651646236" Sep 10 23:51:43.857388 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:43.857348 2583 patch_prober.go:28] interesting pod/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 10 23:51:43.857585 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:43.857422 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" podUID="6de6c23d-6018-4976-9d01-14773a965b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 10 23:51:44.305052 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:44.305021 2583 generic.go:356] "Generic (PLEG): container finished" podID="9357a92b-c1e1-43bb-859a-aa64a9657b1b" containerID="982134d65473497d6f79248f31ddf5da9d3df5224d59938765ad8d94671f0f5f" exitCode=0 Sep 10 23:51:44.305443 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:44.305114 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp" event={"ID":"9357a92b-c1e1-43bb-859a-aa64a9657b1b","Type":"ContainerDied","Data":"982134d65473497d6f79248f31ddf5da9d3df5224d59938765ad8d94671f0f5f"} Sep 10 23:51:44.857134 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:44.857098 2583 patch_prober.go:28] interesting pod/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 10 23:51:44.857312 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:44.857157 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" podUID="6de6c23d-6018-4976-9d01-14773a965b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 10 23:51:45.494650 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:45.494625 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp" Sep 10 23:51:45.608423 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:45.608398 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9357a92b-c1e1-43bb-859a-aa64a9657b1b-maas-api-service-ca\") pod \"9357a92b-c1e1-43bb-859a-aa64a9657b1b\" (UID: \"9357a92b-c1e1-43bb-859a-aa64a9657b1b\") " Sep 10 23:51:45.608709 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:45.608686 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9357a92b-c1e1-43bb-859a-aa64a9657b1b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9357a92b-c1e1-43bb-859a-aa64a9657b1b" (UID: "9357a92b-c1e1-43bb-859a-aa64a9657b1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:51:45.709823 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:45.709782 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9357a92b-c1e1-43bb-859a-aa64a9657b1b-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:45.856547 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:45.856513 2583 patch_prober.go:28] interesting pod/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 10 23:51:45.856708 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:45.856572 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" podUID="6de6c23d-6018-4976-9d01-14773a965b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 10 23:51:46.158323 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:46.158284 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:46.320493 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:46.320459 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp" event={"ID":"9357a92b-c1e1-43bb-859a-aa64a9657b1b","Type":"ContainerDied","Data":"e28a9ca6a3d2bf58fdb9e984bbaf15fd566986f5669fe1be5da9b4d17c1dc579"} Sep 10 23:51:46.320493 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:46.320492 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp" Sep 10 23:51:46.320695 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:46.320505 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e28a9ca6a3d2bf58fdb9e984bbaf15fd566986f5669fe1be5da9b4d17c1dc579" Sep 10 23:51:46.860792 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:46.860758 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:46.861737 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:46.861719 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:49.099382 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:49.099306 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp"] Sep 10 23:51:49.101536 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:49.101512 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9566fcb0-jqwnp"] Sep 10 23:51:49.315031 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:49.314996 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9357a92b-c1e1-43bb-859a-aa64a9657b1b" path="/var/lib/kubelet/pods/9357a92b-c1e1-43bb-859a-aa64a9657b1b/volumes" Sep 10 23:51:50.186838 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:50.186800 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb"] Sep 10 23:51:50.187317 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:50.187037 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" podUID="6de6c23d-6018-4976-9d01-14773a965b30" containerName="istio-proxy" containerID="cri-o://8de43562cc65949168f0ecf4c2a3175d1a32643725675cb1261c4de59b3ab67b" gracePeriod=30 Sep 10 23:51:50.900905 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:50.900871 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl"] Sep 10 23:51:50.901240 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:50.901226 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9357a92b-c1e1-43bb-859a-aa64a9657b1b" containerName="revoke-keys" Sep 10 23:51:50.901303 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:50.901241 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9357a92b-c1e1-43bb-859a-aa64a9657b1b" containerName="revoke-keys" Sep 10 23:51:50.901340 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:50.901315 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9357a92b-c1e1-43bb-859a-aa64a9657b1b" containerName="revoke-keys" Sep 10 23:51:50.904656 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:50.904631 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:50.907187 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:50.907167 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a105b100-openshift-default-dockercfg-sdq8n\"" Sep 10 23:51:50.919866 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:50.919840 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl"] Sep 10 23:51:51.061293 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.061255 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64ca838-ed37-4e69-8d1e-444e10969542-istio-podinfo\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.061461 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.061302 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-istio-envoy\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.061461 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.061372 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-credential-socket\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.061461 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.061408 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-workload-socket\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.061461 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.061429 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-istio-data\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.061461 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.061446 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64ca838-ed37-4e69-8d1e-444e10969542-istiod-ca-cert\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.061629 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.061494 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-workload-certs\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.061629 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.061510 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64ca838-ed37-4e69-8d1e-444e10969542-istio-token\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.061629 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.061532 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vvlg\" (UniqueName: \"kubernetes.io/projected/a64ca838-ed37-4e69-8d1e-444e10969542-kube-api-access-4vvlg\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.162944 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.162855 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64ca838-ed37-4e69-8d1e-444e10969542-istio-podinfo\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.162944 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.162897 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-istio-envoy\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.163188 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.162944 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-credential-socket\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.163188 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.162978 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-workload-socket\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.163188 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.163128 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-istio-data\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.163188 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.163175 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64ca838-ed37-4e69-8d1e-444e10969542-istiod-ca-cert\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.163387 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.163229 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-workload-certs\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.163387 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.163253 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64ca838-ed37-4e69-8d1e-444e10969542-istio-token\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.163387 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.163286 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvlg\" (UniqueName: \"kubernetes.io/projected/a64ca838-ed37-4e69-8d1e-444e10969542-kube-api-access-4vvlg\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.163387 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.163377 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-credential-socket\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.163681 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.163657 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-workload-socket\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.163901 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.163873 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-istio-data\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.164089 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.164041 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64ca838-ed37-4e69-8d1e-444e10969542-istiod-ca-cert\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.164180 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.164138 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-workload-certs\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.165465 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.165444 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-istio-envoy\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.165575 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.165556 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64ca838-ed37-4e69-8d1e-444e10969542-istio-podinfo\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.172384 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.172362 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64ca838-ed37-4e69-8d1e-444e10969542-istio-token\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.172532 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.172509 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vvlg\" (UniqueName: \"kubernetes.io/projected/a64ca838-ed37-4e69-8d1e-444e10969542-kube-api-access-4vvlg\") pod \"e2e-derive-a105b100-openshift-default-b55db675b-2dwjl\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.215399 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.215362 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:51.503908 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.503883 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl"] Sep 10 23:51:51.506064 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:51:51.506038 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64ca838_ed37_4e69_8d1e_444e10969542.slice/crio-fb4e4316079879fc6ff0ece38a30caf0b8ab7eb7e347ccfbe7f4a298042ed957 WatchSource:0}: Error finding container fb4e4316079879fc6ff0ece38a30caf0b8ab7eb7e347ccfbe7f4a298042ed957: Status 404 returned error can't find the container with id fb4e4316079879fc6ff0ece38a30caf0b8ab7eb7e347ccfbe7f4a298042ed957 Sep 10 23:51:51.508310 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.508272 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:51.508412 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.508353 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:51.508412 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:51.508398 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:52.348345 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:52.348309 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" event={"ID":"a64ca838-ed37-4e69-8d1e-444e10969542","Type":"ContainerStarted","Data":"9e6dc6c8d88209b2cc2748693eeb43ef2ce54992ef6a88aad8af358adf504535"} Sep 10 23:51:52.348723 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:52.348362 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" event={"ID":"a64ca838-ed37-4e69-8d1e-444e10969542","Type":"ContainerStarted","Data":"fb4e4316079879fc6ff0ece38a30caf0b8ab7eb7e347ccfbe7f4a298042ed957"} Sep 10 23:51:52.371081 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:52.371029 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" podStartSLOduration=2.371013176 podStartE2EDuration="2.371013176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:52.369766461 +0000 UTC m=+919.672244470" watchObservedRunningTime="2026-09-10 23:51:52.371013176 +0000 UTC m=+919.673491188" Sep 10 23:51:53.215880 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:53.215842 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:53.220823 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:53.220797 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:53.353355 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:53.353324 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:53.354329 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:53.354308 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:51:55.367345 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.367318 2583 generic.go:356] "Generic (PLEG): container finished" podID="6de6c23d-6018-4976-9d01-14773a965b30" containerID="8de43562cc65949168f0ecf4c2a3175d1a32643725675cb1261c4de59b3ab67b" exitCode=0 Sep 10 23:51:55.367730 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.367394 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" event={"ID":"6de6c23d-6018-4976-9d01-14773a965b30","Type":"ContainerDied","Data":"8de43562cc65949168f0ecf4c2a3175d1a32643725675cb1261c4de59b3ab67b"} Sep 10 23:51:55.367730 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.367473 2583 scope.go:117] "RemoveContainer" containerID="8d21f252559ae81f9d95e67c7771b7a322fa75e2dc5a30ef18c1407420e46d39" Sep 10 23:51:55.431306 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.431285 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:55.604331 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.604304 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6pwh\" (UniqueName: \"kubernetes.io/projected/6de6c23d-6018-4976-9d01-14773a965b30-kube-api-access-r6pwh\") pod \"6de6c23d-6018-4976-9d01-14773a965b30\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " Sep 10 23:51:55.604481 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.604346 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de6c23d-6018-4976-9d01-14773a965b30-istiod-ca-cert\") pod \"6de6c23d-6018-4976-9d01-14773a965b30\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " Sep 10 23:51:55.604481 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.604385 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-workload-socket\") pod \"6de6c23d-6018-4976-9d01-14773a965b30\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " Sep 10 23:51:55.604481 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.604424 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de6c23d-6018-4976-9d01-14773a965b30-istio-podinfo\") pod \"6de6c23d-6018-4976-9d01-14773a965b30\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " Sep 10 23:51:55.604661 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.604578 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-credential-socket\") pod \"6de6c23d-6018-4976-9d01-14773a965b30\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " Sep 10 23:51:55.604661 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.604617 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-istio-data\") pod \"6de6c23d-6018-4976-9d01-14773a965b30\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " Sep 10 23:51:55.604661 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.604658 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-istio-envoy\") pod \"6de6c23d-6018-4976-9d01-14773a965b30\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " Sep 10 23:51:55.604804 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.604684 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-workload-certs\") pod \"6de6c23d-6018-4976-9d01-14773a965b30\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " Sep 10 23:51:55.604804 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.604714 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de6c23d-6018-4976-9d01-14773a965b30-istio-token\") pod \"6de6c23d-6018-4976-9d01-14773a965b30\" (UID: \"6de6c23d-6018-4976-9d01-14773a965b30\") " Sep 10 23:51:55.604804 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.604706 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6de6c23d-6018-4976-9d01-14773a965b30" (UID: "6de6c23d-6018-4976-9d01-14773a965b30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:55.604946 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.604843 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6de6c23d-6018-4976-9d01-14773a965b30" (UID: "6de6c23d-6018-4976-9d01-14773a965b30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:55.604946 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.604874 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de6c23d-6018-4976-9d01-14773a965b30-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6de6c23d-6018-4976-9d01-14773a965b30" (UID: "6de6c23d-6018-4976-9d01-14773a965b30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:51:55.604946 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.604908 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-istio-data" (OuterVolumeSpecName: "istio-data") pod "6de6c23d-6018-4976-9d01-14773a965b30" (UID: "6de6c23d-6018-4976-9d01-14773a965b30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:55.605091 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.605056 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:55.605151 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.605107 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:55.605151 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.605122 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de6c23d-6018-4976-9d01-14773a965b30-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:55.605151 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.605136 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:55.605302 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.605151 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6de6c23d-6018-4976-9d01-14773a965b30" (UID: "6de6c23d-6018-4976-9d01-14773a965b30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:55.606621 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.606571 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6de6c23d-6018-4976-9d01-14773a965b30-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6de6c23d-6018-4976-9d01-14773a965b30" (UID: "6de6c23d-6018-4976-9d01-14773a965b30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:51:55.606738 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.606669 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de6c23d-6018-4976-9d01-14773a965b30-kube-api-access-r6pwh" (OuterVolumeSpecName: "kube-api-access-r6pwh") pod "6de6c23d-6018-4976-9d01-14773a965b30" (UID: "6de6c23d-6018-4976-9d01-14773a965b30"). InnerVolumeSpecName "kube-api-access-r6pwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:51:55.606738 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.606673 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6de6c23d-6018-4976-9d01-14773a965b30" (UID: "6de6c23d-6018-4976-9d01-14773a965b30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:55.606958 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.606933 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de6c23d-6018-4976-9d01-14773a965b30-istio-token" (OuterVolumeSpecName: "istio-token") pod "6de6c23d-6018-4976-9d01-14773a965b30" (UID: "6de6c23d-6018-4976-9d01-14773a965b30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:51:55.705685 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.705654 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:55.705685 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.705679 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de6c23d-6018-4976-9d01-14773a965b30-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:55.705685 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.705688 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de6c23d-6018-4976-9d01-14773a965b30-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:55.705900 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.705697 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6pwh\" (UniqueName: \"kubernetes.io/projected/6de6c23d-6018-4976-9d01-14773a965b30-kube-api-access-r6pwh\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:55.705900 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:55.705707 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de6c23d-6018-4976-9d01-14773a965b30-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:51:56.373377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:56.373339 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" event={"ID":"6de6c23d-6018-4976-9d01-14773a965b30","Type":"ContainerDied","Data":"4840213dfa8e8059e8dba6307f7ade51a0010fbb878fa6399816be980051c79d"} Sep 10 23:51:56.373377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:56.373372 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb" Sep 10 23:51:56.373839 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:56.373424 2583 scope.go:117] "RemoveContainer" containerID="8de43562cc65949168f0ecf4c2a3175d1a32643725675cb1261c4de59b3ab67b" Sep 10 23:51:56.398853 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:56.398808 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb"] Sep 10 23:51:56.403065 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:56.403038 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9566fcb0-openshift-default-58cc656658-mmkrb"] Sep 10 23:51:57.236344 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.236308 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm"] Sep 10 23:51:57.236624 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.236613 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de6c23d-6018-4976-9d01-14773a965b30" containerName="istio-proxy" Sep 10 23:51:57.236680 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.236626 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de6c23d-6018-4976-9d01-14773a965b30" containerName="istio-proxy" Sep 10 23:51:57.236680 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.236640 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de6c23d-6018-4976-9d01-14773a965b30" containerName="istio-proxy" Sep 10 23:51:57.236680 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.236646 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de6c23d-6018-4976-9d01-14773a965b30" containerName="istio-proxy" Sep 10 23:51:57.236783 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.236717 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6de6c23d-6018-4976-9d01-14773a965b30" containerName="istio-proxy" Sep 10 23:51:57.240927 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.240906 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" Sep 10 23:51:57.243370 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.243344 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a105b100\"" Sep 10 23:51:57.243559 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.243538 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a105b100-dockercfg-8wvkh\"" Sep 10 23:51:57.248732 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.248701 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm"] Sep 10 23:51:57.310020 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.309983 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6de6c23d-6018-4976-9d01-14773a965b30" path="/var/lib/kubelet/pods/6de6c23d-6018-4976-9d01-14773a965b30/volumes" Sep 10 23:51:57.319810 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.319785 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0e9b0a-4694-41ea-b6d5-4b898654818a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm\" (UID: \"5c0e9b0a-4694-41ea-b6d5-4b898654818a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" Sep 10 23:51:57.319950 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.319814 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbznx\" (UniqueName: \"kubernetes.io/projected/5c0e9b0a-4694-41ea-b6d5-4b898654818a-kube-api-access-qbznx\") pod \"payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm\" (UID: \"5c0e9b0a-4694-41ea-b6d5-4b898654818a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" Sep 10 23:51:57.420831 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.420794 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0e9b0a-4694-41ea-b6d5-4b898654818a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm\" (UID: \"5c0e9b0a-4694-41ea-b6d5-4b898654818a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" Sep 10 23:51:57.420831 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.420835 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbznx\" (UniqueName: \"kubernetes.io/projected/5c0e9b0a-4694-41ea-b6d5-4b898654818a-kube-api-access-qbznx\") pod \"payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm\" (UID: \"5c0e9b0a-4694-41ea-b6d5-4b898654818a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" Sep 10 23:51:57.421560 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.421539 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0e9b0a-4694-41ea-b6d5-4b898654818a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm\" (UID: \"5c0e9b0a-4694-41ea-b6d5-4b898654818a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" Sep 10 23:51:57.428958 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.428937 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbznx\" (UniqueName: \"kubernetes.io/projected/5c0e9b0a-4694-41ea-b6d5-4b898654818a-kube-api-access-qbznx\") pod \"payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm\" (UID: \"5c0e9b0a-4694-41ea-b6d5-4b898654818a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" Sep 10 23:51:57.552853 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.552772 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" Sep 10 23:51:57.822661 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:57.822636 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm"] Sep 10 23:51:57.824738 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:51:57.824710 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0e9b0a_4694_41ea_b6d5_4b898654818a.slice/crio-f88483f4c79632a25bc3126d1e1b1745ec7ccf1a4924ee84239a43384c671695 WatchSource:0}: Error finding container f88483f4c79632a25bc3126d1e1b1745ec7ccf1a4924ee84239a43384c671695: Status 404 returned error can't find the container with id f88483f4c79632a25bc3126d1e1b1745ec7ccf1a4924ee84239a43384c671695 Sep 10 23:51:58.386641 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:51:58.386594 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" event={"ID":"5c0e9b0a-4694-41ea-b6d5-4b898654818a","Type":"ContainerStarted","Data":"f88483f4c79632a25bc3126d1e1b1745ec7ccf1a4924ee84239a43384c671695"} Sep 10 23:52:00.401014 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:00.400974 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" event={"ID":"5c0e9b0a-4694-41ea-b6d5-4b898654818a","Type":"ContainerStarted","Data":"2fee20e927b83247c381c69a0e3efba0436237f71106f5a32bc16b7c9ba9461d"} Sep 10 23:52:00.401394 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:00.401161 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" Sep 10 23:52:00.427802 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:00.427757 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" podStartSLOduration=1.825597769 podStartE2EDuration="3.427743626s" totalImagesPullingTime="1.602145857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:00.427398547 +0000 UTC m=+927.729876558" watchObservedRunningTime="2026-09-10 23:52:00.427743626 +0000 UTC m=+927.730221637" Sep 10 23:52:11.408270 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:11.408235 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" Sep 10 23:52:13.475221 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:13.475190 2583 generic.go:356] "Generic (PLEG): container finished" podID="a64ca838-ed37-4e69-8d1e-444e10969542" containerID="9e6dc6c8d88209b2cc2748693eeb43ef2ce54992ef6a88aad8af358adf504535" exitCode=0 Sep 10 23:52:13.475696 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:13.475267 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" event={"ID":"a64ca838-ed37-4e69-8d1e-444e10969542","Type":"ContainerDied","Data":"9e6dc6c8d88209b2cc2748693eeb43ef2ce54992ef6a88aad8af358adf504535"} Sep 10 23:52:13.475696 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:13.475650 2583 scope.go:117] "RemoveContainer" containerID="9e6dc6c8d88209b2cc2748693eeb43ef2ce54992ef6a88aad8af358adf504535" Sep 10 23:52:13.476677 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:13.476647 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:52:13.476791 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:13.476731 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:52:13.476791 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:13.476785 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:52:14.482583 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:14.482549 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" event={"ID":"a64ca838-ed37-4e69-8d1e-444e10969542","Type":"ContainerStarted","Data":"9b3128dc2a2ea308b5d2b681c04920a5e4f25d24ddd02c5fadee95e43f6945a3"} Sep 10 23:52:15.216503 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:15.216462 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:52:15.216675 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:15.216633 2583 patch_prober.go:28] interesting pod/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 23:52:15.216719 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:15.216684 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" podUID="a64ca838-ed37-4e69-8d1e-444e10969542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 23:52:16.216227 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:16.216187 2583 patch_prober.go:28] interesting pod/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 23:52:16.216694 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:16.216248 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" podUID="a64ca838-ed37-4e69-8d1e-444e10969542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 23:52:17.216284 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:17.216249 2583 patch_prober.go:28] interesting pod/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 23:52:17.216660 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:17.216305 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" podUID="a64ca838-ed37-4e69-8d1e-444e10969542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 23:52:18.215965 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:18.215932 2583 patch_prober.go:28] interesting pod/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 23:52:18.216163 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:18.215985 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" podUID="a64ca838-ed37-4e69-8d1e-444e10969542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 23:52:18.526787 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:18.526665 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9"] Sep 10 23:52:18.527315 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:18.527292 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6de6c23d-6018-4976-9d01-14773a965b30" containerName="istio-proxy" Sep 10 23:52:18.531259 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:18.531232 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9" Sep 10 23:52:18.534204 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:18.534175 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 10 23:52:18.535883 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:18.535862 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 23:52:18.538675 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:18.538653 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9"] Sep 10 23:52:18.627298 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:18.627256 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ee2906-2fb4-40f7-9a63-28eab9074ff7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a105b100-zmgd9\" (UID: \"65ee2906-2fb4-40f7-9a63-28eab9074ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9" Sep 10 23:52:18.728439 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:18.728395 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ee2906-2fb4-40f7-9a63-28eab9074ff7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a105b100-zmgd9\" (UID: \"65ee2906-2fb4-40f7-9a63-28eab9074ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9" Sep 10 23:52:18.729413 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:18.729384 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ee2906-2fb4-40f7-9a63-28eab9074ff7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a105b100-zmgd9\" (UID: \"65ee2906-2fb4-40f7-9a63-28eab9074ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9" Sep 10 23:52:18.849629 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:18.849543 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9" Sep 10 23:52:19.141096 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:19.140723 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9"] Sep 10 23:52:19.155101 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:52:19.153996 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ee2906_2fb4_40f7_9a63_28eab9074ff7.slice/crio-86e2202550d3ac55a6436fc72ec5f227b61efd784e8fa4e1650f9b8678b771a8 WatchSource:0}: Error finding container 86e2202550d3ac55a6436fc72ec5f227b61efd784e8fa4e1650f9b8678b771a8: Status 404 returned error can't find the container with id 86e2202550d3ac55a6436fc72ec5f227b61efd784e8fa4e1650f9b8678b771a8 Sep 10 23:52:19.215803 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:19.215775 2583 patch_prober.go:28] interesting pod/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 23:52:19.215900 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:19.215826 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" podUID="a64ca838-ed37-4e69-8d1e-444e10969542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 23:52:19.507443 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:19.507404 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9" event={"ID":"65ee2906-2fb4-40f7-9a63-28eab9074ff7","Type":"ContainerStarted","Data":"2c64d6dff7a79775dcd197541cc63728b0ff46495ebed22b2016a93e96a0abdf"} Sep 10 23:52:19.507615 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:19.507457 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9" event={"ID":"65ee2906-2fb4-40f7-9a63-28eab9074ff7","Type":"ContainerStarted","Data":"86e2202550d3ac55a6436fc72ec5f227b61efd784e8fa4e1650f9b8678b771a8"} Sep 10 23:52:19.525774 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:19.525722 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9" podStartSLOduration=1.525704292 podStartE2EDuration="1.525704292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:19.523913795 +0000 UTC m=+946.826391807" watchObservedRunningTime="2026-09-10 23:52:19.525704292 +0000 UTC m=+946.828182340" Sep 10 23:52:20.220192 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:20.220115 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:52:20.220616 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:20.220401 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:52:20.221171 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:20.221154 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:52:20.518171 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:20.518055 2583 generic.go:356] "Generic (PLEG): container finished" podID="65ee2906-2fb4-40f7-9a63-28eab9074ff7" containerID="2c64d6dff7a79775dcd197541cc63728b0ff46495ebed22b2016a93e96a0abdf" exitCode=0 Sep 10 23:52:20.518171 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:20.518140 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9" event={"ID":"65ee2906-2fb4-40f7-9a63-28eab9074ff7","Type":"ContainerDied","Data":"2c64d6dff7a79775dcd197541cc63728b0ff46495ebed22b2016a93e96a0abdf"} Sep 10 23:52:21.667429 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:21.667404 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9" Sep 10 23:52:21.768601 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:21.768564 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ee2906-2fb4-40f7-9a63-28eab9074ff7-maas-api-service-ca\") pod \"65ee2906-2fb4-40f7-9a63-28eab9074ff7\" (UID: \"65ee2906-2fb4-40f7-9a63-28eab9074ff7\") " Sep 10 23:52:21.768936 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:21.768911 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ee2906-2fb4-40f7-9a63-28eab9074ff7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65ee2906-2fb4-40f7-9a63-28eab9074ff7" (UID: "65ee2906-2fb4-40f7-9a63-28eab9074ff7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:52:21.869994 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:21.869952 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ee2906-2fb4-40f7-9a63-28eab9074ff7-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:52:22.529545 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:22.529520 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9" Sep 10 23:52:22.529723 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:22.529523 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9" event={"ID":"65ee2906-2fb4-40f7-9a63-28eab9074ff7","Type":"ContainerDied","Data":"86e2202550d3ac55a6436fc72ec5f227b61efd784e8fa4e1650f9b8678b771a8"} Sep 10 23:52:22.529723 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:22.529649 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86e2202550d3ac55a6436fc72ec5f227b61efd784e8fa4e1650f9b8678b771a8" Sep 10 23:52:25.855746 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:25.855706 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm"] Sep 10 23:52:25.856234 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:25.855973 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" podUID="5c0e9b0a-4694-41ea-b6d5-4b898654818a" containerName="payload-pre-processing" containerID="cri-o://2fee20e927b83247c381c69a0e3efba0436237f71106f5a32bc16b7c9ba9461d" gracePeriod=30 Sep 10 23:52:26.014127 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.014092 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9"] Sep 10 23:52:26.016312 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.016285 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a105b100-zmgd9"] Sep 10 23:52:26.138673 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.138487 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" Sep 10 23:52:26.147784 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.147752 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl"] Sep 10 23:52:26.148006 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.147982 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" podUID="a64ca838-ed37-4e69-8d1e-444e10969542" containerName="istio-proxy" containerID="cri-o://9b3128dc2a2ea308b5d2b681c04920a5e4f25d24ddd02c5fadee95e43f6945a3" gracePeriod=30 Sep 10 23:52:26.209641 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.209603 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0e9b0a-4694-41ea-b6d5-4b898654818a-plugins-config-volume\") pod \"5c0e9b0a-4694-41ea-b6d5-4b898654818a\" (UID: \"5c0e9b0a-4694-41ea-b6d5-4b898654818a\") " Sep 10 23:52:26.209810 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.209696 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbznx\" (UniqueName: \"kubernetes.io/projected/5c0e9b0a-4694-41ea-b6d5-4b898654818a-kube-api-access-qbznx\") pod \"5c0e9b0a-4694-41ea-b6d5-4b898654818a\" (UID: \"5c0e9b0a-4694-41ea-b6d5-4b898654818a\") " Sep 10 23:52:26.209973 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.209948 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0e9b0a-4694-41ea-b6d5-4b898654818a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c0e9b0a-4694-41ea-b6d5-4b898654818a" (UID: "5c0e9b0a-4694-41ea-b6d5-4b898654818a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:52:26.211813 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.211775 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0e9b0a-4694-41ea-b6d5-4b898654818a-kube-api-access-qbznx" (OuterVolumeSpecName: "kube-api-access-qbznx") pod "5c0e9b0a-4694-41ea-b6d5-4b898654818a" (UID: "5c0e9b0a-4694-41ea-b6d5-4b898654818a"). InnerVolumeSpecName "kube-api-access-qbznx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:52:26.311143 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.311112 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbznx\" (UniqueName: \"kubernetes.io/projected/5c0e9b0a-4694-41ea-b6d5-4b898654818a-kube-api-access-qbznx\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:52:26.311143 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.311139 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0e9b0a-4694-41ea-b6d5-4b898654818a-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:52:26.556061 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.555965 2583 generic.go:356] "Generic (PLEG): container finished" podID="5c0e9b0a-4694-41ea-b6d5-4b898654818a" containerID="2fee20e927b83247c381c69a0e3efba0436237f71106f5a32bc16b7c9ba9461d" exitCode=0 Sep 10 23:52:26.556061 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.556002 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" Sep 10 23:52:26.556244 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.556050 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" event={"ID":"5c0e9b0a-4694-41ea-b6d5-4b898654818a","Type":"ContainerDied","Data":"2fee20e927b83247c381c69a0e3efba0436237f71106f5a32bc16b7c9ba9461d"} Sep 10 23:52:26.556244 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.556118 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm" event={"ID":"5c0e9b0a-4694-41ea-b6d5-4b898654818a","Type":"ContainerDied","Data":"f88483f4c79632a25bc3126d1e1b1745ec7ccf1a4924ee84239a43384c671695"} Sep 10 23:52:26.556244 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.556148 2583 scope.go:117] "RemoveContainer" containerID="2fee20e927b83247c381c69a0e3efba0436237f71106f5a32bc16b7c9ba9461d" Sep 10 23:52:26.565451 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.565435 2583 scope.go:117] "RemoveContainer" containerID="2fee20e927b83247c381c69a0e3efba0436237f71106f5a32bc16b7c9ba9461d" Sep 10 23:52:26.565705 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:52:26.565688 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fee20e927b83247c381c69a0e3efba0436237f71106f5a32bc16b7c9ba9461d\": container with ID starting with 2fee20e927b83247c381c69a0e3efba0436237f71106f5a32bc16b7c9ba9461d not found: ID does not exist" containerID="2fee20e927b83247c381c69a0e3efba0436237f71106f5a32bc16b7c9ba9461d" Sep 10 23:52:26.565767 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.565716 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fee20e927b83247c381c69a0e3efba0436237f71106f5a32bc16b7c9ba9461d"} err="failed to get container status \"2fee20e927b83247c381c69a0e3efba0436237f71106f5a32bc16b7c9ba9461d\": rpc error: code = NotFound desc = could not find container \"2fee20e927b83247c381c69a0e3efba0436237f71106f5a32bc16b7c9ba9461d\": container with ID starting with 2fee20e927b83247c381c69a0e3efba0436237f71106f5a32bc16b7c9ba9461d not found: ID does not exist" Sep 10 23:52:26.577467 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.577444 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm"] Sep 10 23:52:26.581190 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:26.581169 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a105b100-66c8d4dbc4-ch4hm"] Sep 10 23:52:27.311621 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:27.311584 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c0e9b0a-4694-41ea-b6d5-4b898654818a" path="/var/lib/kubelet/pods/5c0e9b0a-4694-41ea-b6d5-4b898654818a/volumes" Sep 10 23:52:27.312180 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:27.312098 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65ee2906-2fb4-40f7-9a63-28eab9074ff7" path="/var/lib/kubelet/pods/65ee2906-2fb4-40f7-9a63-28eab9074ff7/volumes" Sep 10 23:52:31.396724 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.396700 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:52:31.455177 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455097 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64ca838-ed37-4e69-8d1e-444e10969542-istio-podinfo\") pod \"a64ca838-ed37-4e69-8d1e-444e10969542\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " Sep 10 23:52:31.455177 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455129 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64ca838-ed37-4e69-8d1e-444e10969542-istiod-ca-cert\") pod \"a64ca838-ed37-4e69-8d1e-444e10969542\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " Sep 10 23:52:31.455177 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455154 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-workload-socket\") pod \"a64ca838-ed37-4e69-8d1e-444e10969542\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " Sep 10 23:52:31.455378 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455269 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-istio-envoy\") pod \"a64ca838-ed37-4e69-8d1e-444e10969542\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " Sep 10 23:52:31.455378 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455364 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-credential-socket\") pod \"a64ca838-ed37-4e69-8d1e-444e10969542\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " Sep 10 23:52:31.455452 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455392 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-workload-certs\") pod \"a64ca838-ed37-4e69-8d1e-444e10969542\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " Sep 10 23:52:31.455452 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455397 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a64ca838-ed37-4e69-8d1e-444e10969542" (UID: "a64ca838-ed37-4e69-8d1e-444e10969542"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:31.455452 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455429 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vvlg\" (UniqueName: \"kubernetes.io/projected/a64ca838-ed37-4e69-8d1e-444e10969542-kube-api-access-4vvlg\") pod \"a64ca838-ed37-4e69-8d1e-444e10969542\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " Sep 10 23:52:31.455579 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455488 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-istio-data\") pod \"a64ca838-ed37-4e69-8d1e-444e10969542\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " Sep 10 23:52:31.455579 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455534 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64ca838-ed37-4e69-8d1e-444e10969542-istio-token\") pod \"a64ca838-ed37-4e69-8d1e-444e10969542\" (UID: \"a64ca838-ed37-4e69-8d1e-444e10969542\") " Sep 10 23:52:31.455685 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455605 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64ca838-ed37-4e69-8d1e-444e10969542-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a64ca838-ed37-4e69-8d1e-444e10969542" (UID: "a64ca838-ed37-4e69-8d1e-444e10969542"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:52:31.455685 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455624 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a64ca838-ed37-4e69-8d1e-444e10969542" (UID: "a64ca838-ed37-4e69-8d1e-444e10969542"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:31.455685 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455614 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a64ca838-ed37-4e69-8d1e-444e10969542" (UID: "a64ca838-ed37-4e69-8d1e-444e10969542"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:31.455843 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455752 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-istio-data" (OuterVolumeSpecName: "istio-data") pod "a64ca838-ed37-4e69-8d1e-444e10969542" (UID: "a64ca838-ed37-4e69-8d1e-444e10969542"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:31.455911 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455885 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64ca838-ed37-4e69-8d1e-444e10969542-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:52:31.455911 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455905 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:52:31.455911 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455913 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:52:31.456091 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455923 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:52:31.456091 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.455931 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:52:31.457422 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.457397 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a64ca838-ed37-4e69-8d1e-444e10969542-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a64ca838-ed37-4e69-8d1e-444e10969542" (UID: "a64ca838-ed37-4e69-8d1e-444e10969542"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:52:31.457546 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.457528 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a64ca838-ed37-4e69-8d1e-444e10969542" (UID: "a64ca838-ed37-4e69-8d1e-444e10969542"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:31.457590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.457554 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64ca838-ed37-4e69-8d1e-444e10969542-istio-token" (OuterVolumeSpecName: "istio-token") pod "a64ca838-ed37-4e69-8d1e-444e10969542" (UID: "a64ca838-ed37-4e69-8d1e-444e10969542"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:52:31.457590 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.457569 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64ca838-ed37-4e69-8d1e-444e10969542-kube-api-access-4vvlg" (OuterVolumeSpecName: "kube-api-access-4vvlg") pod "a64ca838-ed37-4e69-8d1e-444e10969542" (UID: "a64ca838-ed37-4e69-8d1e-444e10969542"). InnerVolumeSpecName "kube-api-access-4vvlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:52:31.556896 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.556860 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64ca838-ed37-4e69-8d1e-444e10969542-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:52:31.556896 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.556889 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64ca838-ed37-4e69-8d1e-444e10969542-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:52:31.556896 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.556899 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vvlg\" (UniqueName: \"kubernetes.io/projected/a64ca838-ed37-4e69-8d1e-444e10969542-kube-api-access-4vvlg\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:52:31.556896 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.556910 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64ca838-ed37-4e69-8d1e-444e10969542-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:52:31.581742 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.581706 2583 generic.go:356] "Generic (PLEG): container finished" podID="a64ca838-ed37-4e69-8d1e-444e10969542" containerID="9b3128dc2a2ea308b5d2b681c04920a5e4f25d24ddd02c5fadee95e43f6945a3" exitCode=0 Sep 10 23:52:31.581857 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.581754 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" Sep 10 23:52:31.581857 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.581792 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" event={"ID":"a64ca838-ed37-4e69-8d1e-444e10969542","Type":"ContainerDied","Data":"9b3128dc2a2ea308b5d2b681c04920a5e4f25d24ddd02c5fadee95e43f6945a3"} Sep 10 23:52:31.581857 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.581851 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl" event={"ID":"a64ca838-ed37-4e69-8d1e-444e10969542","Type":"ContainerDied","Data":"fb4e4316079879fc6ff0ece38a30caf0b8ab7eb7e347ccfbe7f4a298042ed957"} Sep 10 23:52:31.581979 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.581883 2583 scope.go:117] "RemoveContainer" containerID="9b3128dc2a2ea308b5d2b681c04920a5e4f25d24ddd02c5fadee95e43f6945a3" Sep 10 23:52:31.591315 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.591299 2583 scope.go:117] "RemoveContainer" containerID="9e6dc6c8d88209b2cc2748693eeb43ef2ce54992ef6a88aad8af358adf504535" Sep 10 23:52:31.604131 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.604110 2583 scope.go:117] "RemoveContainer" containerID="9b3128dc2a2ea308b5d2b681c04920a5e4f25d24ddd02c5fadee95e43f6945a3" Sep 10 23:52:31.604440 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:52:31.604420 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3128dc2a2ea308b5d2b681c04920a5e4f25d24ddd02c5fadee95e43f6945a3\": container with ID starting with 9b3128dc2a2ea308b5d2b681c04920a5e4f25d24ddd02c5fadee95e43f6945a3 not found: ID does not exist" containerID="9b3128dc2a2ea308b5d2b681c04920a5e4f25d24ddd02c5fadee95e43f6945a3" Sep 10 23:52:31.604494 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.604449 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3128dc2a2ea308b5d2b681c04920a5e4f25d24ddd02c5fadee95e43f6945a3"} err="failed to get container status \"9b3128dc2a2ea308b5d2b681c04920a5e4f25d24ddd02c5fadee95e43f6945a3\": rpc error: code = NotFound desc = could not find container \"9b3128dc2a2ea308b5d2b681c04920a5e4f25d24ddd02c5fadee95e43f6945a3\": container with ID starting with 9b3128dc2a2ea308b5d2b681c04920a5e4f25d24ddd02c5fadee95e43f6945a3 not found: ID does not exist" Sep 10 23:52:31.604494 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.604468 2583 scope.go:117] "RemoveContainer" containerID="9e6dc6c8d88209b2cc2748693eeb43ef2ce54992ef6a88aad8af358adf504535" Sep 10 23:52:31.604707 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:52:31.604688 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6dc6c8d88209b2cc2748693eeb43ef2ce54992ef6a88aad8af358adf504535\": container with ID starting with 9e6dc6c8d88209b2cc2748693eeb43ef2ce54992ef6a88aad8af358adf504535 not found: ID does not exist" containerID="9e6dc6c8d88209b2cc2748693eeb43ef2ce54992ef6a88aad8af358adf504535" Sep 10 23:52:31.604751 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.604711 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6dc6c8d88209b2cc2748693eeb43ef2ce54992ef6a88aad8af358adf504535"} err="failed to get container status \"9e6dc6c8d88209b2cc2748693eeb43ef2ce54992ef6a88aad8af358adf504535\": rpc error: code = NotFound desc = could not find container \"9e6dc6c8d88209b2cc2748693eeb43ef2ce54992ef6a88aad8af358adf504535\": container with ID starting with 9e6dc6c8d88209b2cc2748693eeb43ef2ce54992ef6a88aad8af358adf504535 not found: ID does not exist" Sep 10 23:52:31.610795 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.610765 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl"] Sep 10 23:52:31.614517 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:31.614497 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a105b100-openshift-default-b55db675b-2dwjl"] Sep 10 23:52:33.308256 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:33.308224 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a64ca838-ed37-4e69-8d1e-444e10969542" path="/var/lib/kubelet/pods/a64ca838-ed37-4e69-8d1e-444e10969542/volumes" Sep 10 23:52:41.556388 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.556347 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9"] Sep 10 23:52:41.556763 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.556671 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c0e9b0a-4694-41ea-b6d5-4b898654818a" containerName="payload-pre-processing" Sep 10 23:52:41.556763 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.556684 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c0e9b0a-4694-41ea-b6d5-4b898654818a" containerName="payload-pre-processing" Sep 10 23:52:41.556763 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.556705 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a64ca838-ed37-4e69-8d1e-444e10969542" containerName="istio-proxy" Sep 10 23:52:41.556763 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.556710 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a64ca838-ed37-4e69-8d1e-444e10969542" containerName="istio-proxy" Sep 10 23:52:41.556763 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.556720 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ee2906-2fb4-40f7-9a63-28eab9074ff7" containerName="revoke-keys" Sep 10 23:52:41.556763 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.556725 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ee2906-2fb4-40f7-9a63-28eab9074ff7" containerName="revoke-keys" Sep 10 23:52:41.556763 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.556732 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a64ca838-ed37-4e69-8d1e-444e10969542" containerName="istio-proxy" Sep 10 23:52:41.556763 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.556737 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a64ca838-ed37-4e69-8d1e-444e10969542" containerName="istio-proxy" Sep 10 23:52:41.557017 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.556784 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a64ca838-ed37-4e69-8d1e-444e10969542" containerName="istio-proxy" Sep 10 23:52:41.557017 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.556793 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c0e9b0a-4694-41ea-b6d5-4b898654818a" containerName="payload-pre-processing" Sep 10 23:52:41.557017 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.556800 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a64ca838-ed37-4e69-8d1e-444e10969542" containerName="istio-proxy" Sep 10 23:52:41.557017 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.556807 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ee2906-2fb4-40f7-9a63-28eab9074ff7" containerName="revoke-keys" Sep 10 23:52:41.560909 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.560892 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9" Sep 10 23:52:41.563498 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.563474 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 23:52:41.563498 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.563481 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 10 23:52:41.568834 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.568330 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9"] Sep 10 23:52:41.643485 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.643449 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b64d76f-332a-499d-a83d-19f59f1633e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b3de77-f27b9\" (UID: \"6b64d76f-332a-499d-a83d-19f59f1633e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9" Sep 10 23:52:41.743931 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.743893 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b64d76f-332a-499d-a83d-19f59f1633e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b3de77-f27b9\" (UID: \"6b64d76f-332a-499d-a83d-19f59f1633e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9" Sep 10 23:52:41.744549 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.744527 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b64d76f-332a-499d-a83d-19f59f1633e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b3de77-f27b9\" (UID: \"6b64d76f-332a-499d-a83d-19f59f1633e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9" Sep 10 23:52:41.872713 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:41.872682 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9" Sep 10 23:52:42.140799 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:42.140689 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9"] Sep 10 23:52:42.143635 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:52:42.143587 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b64d76f_332a_499d_a83d_19f59f1633e2.slice/crio-fd12eb189ab037cdc1b454d62178dbe6bea31542e74a2804ca33909c511039b9 WatchSource:0}: Error finding container fd12eb189ab037cdc1b454d62178dbe6bea31542e74a2804ca33909c511039b9: Status 404 returned error can't find the container with id fd12eb189ab037cdc1b454d62178dbe6bea31542e74a2804ca33909c511039b9 Sep 10 23:52:42.635465 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:42.635432 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9" event={"ID":"6b64d76f-332a-499d-a83d-19f59f1633e2","Type":"ContainerStarted","Data":"2e1f3bcadd655f38dc8632a46b29dda579c0ec071a9ba9140ea5a2ad04728591"} Sep 10 23:52:42.635843 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:42.635482 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9" event={"ID":"6b64d76f-332a-499d-a83d-19f59f1633e2","Type":"ContainerStarted","Data":"fd12eb189ab037cdc1b454d62178dbe6bea31542e74a2804ca33909c511039b9"} Sep 10 23:52:42.662550 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:42.662501 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9" podStartSLOduration=1.662484875 podStartE2EDuration="1.662484875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:42.661485837 +0000 UTC m=+969.963963850" watchObservedRunningTime="2026-09-10 23:52:42.662484875 +0000 UTC m=+969.964962887" Sep 10 23:52:43.641405 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:43.641369 2583 generic.go:356] "Generic (PLEG): container finished" podID="6b64d76f-332a-499d-a83d-19f59f1633e2" containerID="2e1f3bcadd655f38dc8632a46b29dda579c0ec071a9ba9140ea5a2ad04728591" exitCode=0 Sep 10 23:52:43.641773 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:43.641411 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9" event={"ID":"6b64d76f-332a-499d-a83d-19f59f1633e2","Type":"ContainerDied","Data":"2e1f3bcadd655f38dc8632a46b29dda579c0ec071a9ba9140ea5a2ad04728591"} Sep 10 23:52:44.772931 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:44.772907 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9" Sep 10 23:52:44.869631 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:44.869599 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b64d76f-332a-499d-a83d-19f59f1633e2-maas-api-service-ca\") pod \"6b64d76f-332a-499d-a83d-19f59f1633e2\" (UID: \"6b64d76f-332a-499d-a83d-19f59f1633e2\") " Sep 10 23:52:44.869937 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:44.869913 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b64d76f-332a-499d-a83d-19f59f1633e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6b64d76f-332a-499d-a83d-19f59f1633e2" (UID: "6b64d76f-332a-499d-a83d-19f59f1633e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:52:44.971016 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:44.970938 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b64d76f-332a-499d-a83d-19f59f1633e2-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:52:45.652271 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:45.652240 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9" Sep 10 23:52:45.652439 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:45.652272 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9" event={"ID":"6b64d76f-332a-499d-a83d-19f59f1633e2","Type":"ContainerDied","Data":"fd12eb189ab037cdc1b454d62178dbe6bea31542e74a2804ca33909c511039b9"} Sep 10 23:52:45.652439 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:45.652323 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd12eb189ab037cdc1b454d62178dbe6bea31542e74a2804ca33909c511039b9" Sep 10 23:52:52.329695 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:52.329624 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9"] Sep 10 23:52:52.331971 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:52.331941 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3de77-f27b9"] Sep 10 23:52:53.307578 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:53.307540 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b64d76f-332a-499d-a83d-19f59f1633e2" path="/var/lib/kubelet/pods/6b64d76f-332a-499d-a83d-19f59f1633e2/volumes" Sep 10 23:52:55.224155 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.224118 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74"] Sep 10 23:52:55.224661 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.224449 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b64d76f-332a-499d-a83d-19f59f1633e2" containerName="revoke-keys" Sep 10 23:52:55.224661 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.224460 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b64d76f-332a-499d-a83d-19f59f1633e2" containerName="revoke-keys" Sep 10 23:52:55.224661 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.224514 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b64d76f-332a-499d-a83d-19f59f1633e2" containerName="revoke-keys" Sep 10 23:52:55.227580 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.227561 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.229756 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.229735 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-ca420b8d-kserve-self-signed-certs\"" Sep 10 23:52:55.237628 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.237604 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74"] Sep 10 23:52:55.360929 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.360897 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-model-cache\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.361120 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.360936 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-dshm\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.361120 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.361011 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-home\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.361120 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.361044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gtfw\" (UniqueName: \"kubernetes.io/projected/2cb5c73b-a237-475e-92b1-6b7e04517619-kube-api-access-7gtfw\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.361120 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.361065 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-tmp-dir\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.361120 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.361093 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb5c73b-a237-475e-92b1-6b7e04517619-tls-certs\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.461930 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.461895 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-home\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.462115 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.461945 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gtfw\" (UniqueName: \"kubernetes.io/projected/2cb5c73b-a237-475e-92b1-6b7e04517619-kube-api-access-7gtfw\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.462115 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.461968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-tmp-dir\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.462115 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.461984 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb5c73b-a237-475e-92b1-6b7e04517619-tls-certs\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.462115 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.462011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-model-cache\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.462115 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.462034 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-dshm\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.462385 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.462363 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-home\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.462427 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.462373 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-tmp-dir\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.462508 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.462492 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-model-cache\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.464297 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.464274 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-dshm\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.464515 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.464492 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb5c73b-a237-475e-92b1-6b7e04517619-tls-certs\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.472883 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.471743 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gtfw\" (UniqueName: \"kubernetes.io/projected/2cb5c73b-a237-475e-92b1-6b7e04517619-kube-api-access-7gtfw\") pod \"e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.536411 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.536333 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:52:55.809201 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:55.809176 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74"] Sep 10 23:52:55.811250 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:52:55.811226 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cb5c73b_a237_475e_92b1_6b7e04517619.slice/crio-9e310ba56341cbf28301ff7e5836160f233abb5ddc6e570a65bc2ffb8049ba09 WatchSource:0}: Error finding container 9e310ba56341cbf28301ff7e5836160f233abb5ddc6e570a65bc2ffb8049ba09: Status 404 returned error can't find the container with id 9e310ba56341cbf28301ff7e5836160f233abb5ddc6e570a65bc2ffb8049ba09 Sep 10 23:52:56.707081 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:56.707043 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" event={"ID":"2cb5c73b-a237-475e-92b1-6b7e04517619","Type":"ContainerStarted","Data":"4817c4fd28b5d7a264906f4107970e16710bac95324198e3002ef47d17a9441b"} Sep 10 23:52:56.707480 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:56.707103 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" event={"ID":"2cb5c73b-a237-475e-92b1-6b7e04517619","Type":"ContainerStarted","Data":"9e310ba56341cbf28301ff7e5836160f233abb5ddc6e570a65bc2ffb8049ba09"} Sep 10 23:52:56.723422 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:52:56.723377 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" podStartSLOduration=1.723361215 podStartE2EDuration="1.723361215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:56.721604012 +0000 UTC m=+984.024082023" watchObservedRunningTime="2026-09-10 23:52:56.723361215 +0000 UTC m=+984.025839226" Sep 10 23:53:05.537245 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:05.537207 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:53:05.537693 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:05.537264 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:53:05.549725 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:05.549692 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:53:05.760330 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:05.760297 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:53:35.309361 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:35.309328 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz"] Sep 10 23:53:35.309955 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:35.309535 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" containerID="cri-o://78ce22a029a1d2266ce6dc29a121c29ece0084fc60b1f2a1ab8a454bc280f5a9" gracePeriod=30 Sep 10 23:53:36.159531 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.159496 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8"] Sep 10 23:53:36.163272 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.163245 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.166485 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.166459 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-7bb7cb-openshift-default-dockercfg-znj45\"" Sep 10 23:53:36.176406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.176378 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8"] Sep 10 23:53:36.207594 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.207559 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-podinfo\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.207755 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.207598 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-data\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.207755 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.207624 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-token\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.207755 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.207670 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-workload-socket\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.207755 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.207693 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-envoy\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.207951 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.207828 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vp2q\" (UniqueName: \"kubernetes.io/projected/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-kube-api-access-5vp2q\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.207951 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.207881 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-workload-certs\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.208028 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.207946 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-credential-socket\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.208028 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.208010 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istiod-ca-cert\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.310560 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.310517 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vp2q\" (UniqueName: \"kubernetes.io/projected/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-kube-api-access-5vp2q\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.311561 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.311532 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-workload-certs\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.312199 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.312100 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-workload-certs\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.312372 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.312355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-credential-socket\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.312768 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.312745 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istiod-ca-cert\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.313611 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.313593 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-podinfo\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.314137 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.314119 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-data\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.314253 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.312927 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-credential-socket\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.314348 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.313510 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istiod-ca-cert\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.314666 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.314649 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-data\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.314788 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.314773 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-token\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.314921 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.314905 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-workload-socket\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.315012 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.315000 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-envoy\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.316003 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.315979 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-workload-socket\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.318992 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.318893 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-envoy\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.319208 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.319188 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-podinfo\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.320534 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.320509 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vp2q\" (UniqueName: \"kubernetes.io/projected/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-kube-api-access-5vp2q\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.323539 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.323508 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-token\") pod \"e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.473933 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.473838 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:36.752443 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.752415 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8"] Sep 10 23:53:36.754602 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:53:36.754570 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d52fe77_353b_4e07_a7b1_79ee73baa3c6.slice/crio-d97b5eb2afa6a7cb5e7d9cdd931adc1403d3855a1b54e607137bf82592ad0248 WatchSource:0}: Error finding container d97b5eb2afa6a7cb5e7d9cdd931adc1403d3855a1b54e607137bf82592ad0248: Status 404 returned error can't find the container with id d97b5eb2afa6a7cb5e7d9cdd931adc1403d3855a1b54e607137bf82592ad0248 Sep 10 23:53:36.756890 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.756860 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:53:36.756971 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.756925 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:53:36.756971 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.756954 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:53:36.893410 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.893375 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" event={"ID":"6d52fe77-353b-4e07-a7b1-79ee73baa3c6","Type":"ContainerStarted","Data":"03dd36eaa0c46c6fadace282efcb1dc3ea413b608f0d2ce103b3770d87c21292"} Sep 10 23:53:36.893583 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.893440 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" event={"ID":"6d52fe77-353b-4e07-a7b1-79ee73baa3c6","Type":"ContainerStarted","Data":"d97b5eb2afa6a7cb5e7d9cdd931adc1403d3855a1b54e607137bf82592ad0248"} Sep 10 23:53:36.914976 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:36.914922 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" podStartSLOduration=0.91490259 podStartE2EDuration="914.90259ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:53:36.913865685 +0000 UTC m=+1024.216343693" watchObservedRunningTime="2026-09-10 23:53:36.91490259 +0000 UTC m=+1024.217380602" Sep 10 23:53:37.474337 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:37.474299 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:37.479110 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:37.479083 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:37.898541 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:37.898508 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:37.899525 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:37.899506 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:40.559939 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.559913 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:53:40.655162 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655128 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-credential-socket\") pod \"dbdd3267-3062-4b90-a272-1bfa4798984a\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " Sep 10 23:53:40.655162 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655165 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-envoy\") pod \"dbdd3267-3062-4b90-a272-1bfa4798984a\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " Sep 10 23:53:40.655403 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655191 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-data\") pod \"dbdd3267-3062-4b90-a272-1bfa4798984a\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " Sep 10 23:53:40.655403 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655232 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpzxc\" (UniqueName: \"kubernetes.io/projected/dbdd3267-3062-4b90-a272-1bfa4798984a-kube-api-access-kpzxc\") pod \"dbdd3267-3062-4b90-a272-1bfa4798984a\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " Sep 10 23:53:40.655403 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655277 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-workload-certs\") pod \"dbdd3267-3062-4b90-a272-1bfa4798984a\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " Sep 10 23:53:40.655403 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655348 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-workload-socket\") pod \"dbdd3267-3062-4b90-a272-1bfa4798984a\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " Sep 10 23:53:40.655403 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655371 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-token\") pod \"dbdd3267-3062-4b90-a272-1bfa4798984a\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " Sep 10 23:53:40.655662 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655421 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-podinfo\") pod \"dbdd3267-3062-4b90-a272-1bfa4798984a\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " Sep 10 23:53:40.655662 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655461 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdd3267-3062-4b90-a272-1bfa4798984a-istiod-ca-cert\") pod \"dbdd3267-3062-4b90-a272-1bfa4798984a\" (UID: \"dbdd3267-3062-4b90-a272-1bfa4798984a\") " Sep 10 23:53:40.655662 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655497 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dbdd3267-3062-4b90-a272-1bfa4798984a" (UID: "dbdd3267-3062-4b90-a272-1bfa4798984a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:53:40.655832 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655697 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dbdd3267-3062-4b90-a272-1bfa4798984a" (UID: "dbdd3267-3062-4b90-a272-1bfa4798984a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:53:40.655832 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655711 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dbdd3267-3062-4b90-a272-1bfa4798984a" (UID: "dbdd3267-3062-4b90-a272-1bfa4798984a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:53:40.655832 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655810 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:53:40.655832 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655832 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:53:40.656033 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655845 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:53:40.656033 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.655930 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-data" (OuterVolumeSpecName: "istio-data") pod "dbdd3267-3062-4b90-a272-1bfa4798984a" (UID: "dbdd3267-3062-4b90-a272-1bfa4798984a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:53:40.656177 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.656149 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbdd3267-3062-4b90-a272-1bfa4798984a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dbdd3267-3062-4b90-a272-1bfa4798984a" (UID: "dbdd3267-3062-4b90-a272-1bfa4798984a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:53:40.657525 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.657498 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dbdd3267-3062-4b90-a272-1bfa4798984a" (UID: "dbdd3267-3062-4b90-a272-1bfa4798984a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:53:40.657525 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.657516 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-token" (OuterVolumeSpecName: "istio-token") pod "dbdd3267-3062-4b90-a272-1bfa4798984a" (UID: "dbdd3267-3062-4b90-a272-1bfa4798984a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:53:40.657664 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.657536 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dbdd3267-3062-4b90-a272-1bfa4798984a" (UID: "dbdd3267-3062-4b90-a272-1bfa4798984a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:53:40.657664 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.657574 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdd3267-3062-4b90-a272-1bfa4798984a-kube-api-access-kpzxc" (OuterVolumeSpecName: "kube-api-access-kpzxc") pod "dbdd3267-3062-4b90-a272-1bfa4798984a" (UID: "dbdd3267-3062-4b90-a272-1bfa4798984a"). InnerVolumeSpecName "kube-api-access-kpzxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:53:40.756961 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.756930 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:53:40.756961 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.756954 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:53:40.756961 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.756963 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbdd3267-3062-4b90-a272-1bfa4798984a-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:53:40.757180 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.756972 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:53:40.757180 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.756980 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbdd3267-3062-4b90-a272-1bfa4798984a-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:53:40.757180 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.756988 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpzxc\" (UniqueName: \"kubernetes.io/projected/dbdd3267-3062-4b90-a272-1bfa4798984a-kube-api-access-kpzxc\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:53:40.915896 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.915808 2583 generic.go:356] "Generic (PLEG): container finished" podID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerID="78ce22a029a1d2266ce6dc29a121c29ece0084fc60b1f2a1ab8a454bc280f5a9" exitCode=0 Sep 10 23:53:40.915896 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.915856 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" Sep 10 23:53:40.916061 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.915894 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" event={"ID":"dbdd3267-3062-4b90-a272-1bfa4798984a","Type":"ContainerDied","Data":"78ce22a029a1d2266ce6dc29a121c29ece0084fc60b1f2a1ab8a454bc280f5a9"} Sep 10 23:53:40.916061 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.915982 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz" event={"ID":"dbdd3267-3062-4b90-a272-1bfa4798984a","Type":"ContainerDied","Data":"2dd1cd38394c576cc22a8291bb575c13ffbc86f8a742f7507210d54f779272bc"} Sep 10 23:53:40.916061 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.916012 2583 scope.go:117] "RemoveContainer" containerID="78ce22a029a1d2266ce6dc29a121c29ece0084fc60b1f2a1ab8a454bc280f5a9" Sep 10 23:53:40.924867 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.924849 2583 scope.go:117] "RemoveContainer" containerID="bf859e29274b1ef820f0fbd99fd42081eeedcc094d813bf352d823c0f82a784a" Sep 10 23:53:40.932385 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.932369 2583 scope.go:117] "RemoveContainer" containerID="78ce22a029a1d2266ce6dc29a121c29ece0084fc60b1f2a1ab8a454bc280f5a9" Sep 10 23:53:40.932664 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:53:40.932646 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ce22a029a1d2266ce6dc29a121c29ece0084fc60b1f2a1ab8a454bc280f5a9\": container with ID starting with 78ce22a029a1d2266ce6dc29a121c29ece0084fc60b1f2a1ab8a454bc280f5a9 not found: ID does not exist" containerID="78ce22a029a1d2266ce6dc29a121c29ece0084fc60b1f2a1ab8a454bc280f5a9" Sep 10 23:53:40.932721 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.932672 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ce22a029a1d2266ce6dc29a121c29ece0084fc60b1f2a1ab8a454bc280f5a9"} err="failed to get container status \"78ce22a029a1d2266ce6dc29a121c29ece0084fc60b1f2a1ab8a454bc280f5a9\": rpc error: code = NotFound desc = could not find container \"78ce22a029a1d2266ce6dc29a121c29ece0084fc60b1f2a1ab8a454bc280f5a9\": container with ID starting with 78ce22a029a1d2266ce6dc29a121c29ece0084fc60b1f2a1ab8a454bc280f5a9 not found: ID does not exist" Sep 10 23:53:40.932721 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.932691 2583 scope.go:117] "RemoveContainer" containerID="bf859e29274b1ef820f0fbd99fd42081eeedcc094d813bf352d823c0f82a784a" Sep 10 23:53:40.932911 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:53:40.932895 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf859e29274b1ef820f0fbd99fd42081eeedcc094d813bf352d823c0f82a784a\": container with ID starting with bf859e29274b1ef820f0fbd99fd42081eeedcc094d813bf352d823c0f82a784a not found: ID does not exist" containerID="bf859e29274b1ef820f0fbd99fd42081eeedcc094d813bf352d823c0f82a784a" Sep 10 23:53:40.932959 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.932916 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf859e29274b1ef820f0fbd99fd42081eeedcc094d813bf352d823c0f82a784a"} err="failed to get container status \"bf859e29274b1ef820f0fbd99fd42081eeedcc094d813bf352d823c0f82a784a\": rpc error: code = NotFound desc = could not find container \"bf859e29274b1ef820f0fbd99fd42081eeedcc094d813bf352d823c0f82a784a\": container with ID starting with bf859e29274b1ef820f0fbd99fd42081eeedcc094d813bf352d823c0f82a784a not found: ID does not exist" Sep 10 23:53:40.942472 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.942444 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz"] Sep 10 23:53:40.947497 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:40.947472 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-b3de77-openshift-default-6794588794-bqjsz"] Sep 10 23:53:41.308593 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.308501 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" path="/var/lib/kubelet/pods/dbdd3267-3062-4b90-a272-1bfa4798984a/volumes" Sep 10 23:53:41.751352 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.751319 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n"] Sep 10 23:53:41.751730 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.751650 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" Sep 10 23:53:41.751730 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.751664 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" Sep 10 23:53:41.751730 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.751728 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" Sep 10 23:53:41.751831 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.751739 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" Sep 10 23:53:41.756261 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.756244 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" Sep 10 23:53:41.758819 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.758796 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-7bb7cb-dockercfg-k247c\"" Sep 10 23:53:41.758918 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.758847 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-7bb7cb\"" Sep 10 23:53:41.763701 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.763676 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6"] Sep 10 23:53:41.763965 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.763951 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" Sep 10 23:53:41.763965 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.763963 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbdd3267-3062-4b90-a272-1bfa4798984a" containerName="istio-proxy" Sep 10 23:53:41.766757 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.766736 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" Sep 10 23:53:41.768221 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.768198 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n"] Sep 10 23:53:41.777883 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.777861 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6"] Sep 10 23:53:41.866929 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.866887 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwcph\" (UniqueName: \"kubernetes.io/projected/a78a8a0e-80df-4ab3-b2af-eab9a3268037-kube-api-access-hwcph\") pod \"payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6\" (UID: \"a78a8a0e-80df-4ab3-b2af-eab9a3268037\") " pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" Sep 10 23:53:41.867128 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.866997 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj54v\" (UniqueName: \"kubernetes.io/projected/374ad816-6d4b-4199-bdaa-47a58d95f543-kube-api-access-gj54v\") pod \"payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n\" (UID: \"374ad816-6d4b-4199-bdaa-47a58d95f543\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" Sep 10 23:53:41.867128 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.867027 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78a8a0e-80df-4ab3-b2af-eab9a3268037-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6\" (UID: \"a78a8a0e-80df-4ab3-b2af-eab9a3268037\") " pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" Sep 10 23:53:41.867128 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.867103 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/374ad816-6d4b-4199-bdaa-47a58d95f543-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n\" (UID: \"374ad816-6d4b-4199-bdaa-47a58d95f543\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" Sep 10 23:53:41.967989 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.967953 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwcph\" (UniqueName: \"kubernetes.io/projected/a78a8a0e-80df-4ab3-b2af-eab9a3268037-kube-api-access-hwcph\") pod \"payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6\" (UID: \"a78a8a0e-80df-4ab3-b2af-eab9a3268037\") " pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" Sep 10 23:53:41.968179 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.968055 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj54v\" (UniqueName: \"kubernetes.io/projected/374ad816-6d4b-4199-bdaa-47a58d95f543-kube-api-access-gj54v\") pod \"payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n\" (UID: \"374ad816-6d4b-4199-bdaa-47a58d95f543\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" Sep 10 23:53:41.968179 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.968106 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78a8a0e-80df-4ab3-b2af-eab9a3268037-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6\" (UID: \"a78a8a0e-80df-4ab3-b2af-eab9a3268037\") " pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" Sep 10 23:53:41.968179 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.968151 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/374ad816-6d4b-4199-bdaa-47a58d95f543-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n\" (UID: \"374ad816-6d4b-4199-bdaa-47a58d95f543\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" Sep 10 23:53:41.968853 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.968831 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78a8a0e-80df-4ab3-b2af-eab9a3268037-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6\" (UID: \"a78a8a0e-80df-4ab3-b2af-eab9a3268037\") " pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" Sep 10 23:53:41.968903 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.968884 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/374ad816-6d4b-4199-bdaa-47a58d95f543-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n\" (UID: \"374ad816-6d4b-4199-bdaa-47a58d95f543\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" Sep 10 23:53:41.975942 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.975912 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj54v\" (UniqueName: \"kubernetes.io/projected/374ad816-6d4b-4199-bdaa-47a58d95f543-kube-api-access-gj54v\") pod \"payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n\" (UID: \"374ad816-6d4b-4199-bdaa-47a58d95f543\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" Sep 10 23:53:41.976228 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:41.976210 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwcph\" (UniqueName: \"kubernetes.io/projected/a78a8a0e-80df-4ab3-b2af-eab9a3268037-kube-api-access-hwcph\") pod \"payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6\" (UID: \"a78a8a0e-80df-4ab3-b2af-eab9a3268037\") " pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" Sep 10 23:53:42.066953 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:42.066874 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" Sep 10 23:53:42.080487 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:42.080465 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" Sep 10 23:53:42.345085 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:42.345037 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n"] Sep 10 23:53:42.346713 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:53:42.346686 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374ad816_6d4b_4199_bdaa_47a58d95f543.slice/crio-f0864bfb96bf0508b35d2fb9d548eb23d23deff32533973d165902af34dbea94 WatchSource:0}: Error finding container f0864bfb96bf0508b35d2fb9d548eb23d23deff32533973d165902af34dbea94: Status 404 returned error can't find the container with id f0864bfb96bf0508b35d2fb9d548eb23d23deff32533973d165902af34dbea94 Sep 10 23:53:42.361920 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:42.361849 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6"] Sep 10 23:53:42.364174 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:53:42.364149 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78a8a0e_80df_4ab3_b2af_eab9a3268037.slice/crio-e35bceb916fe1e238994d84de9fc8da95ea3943a264f4c57bcc1b6553bc832f4 WatchSource:0}: Error finding container e35bceb916fe1e238994d84de9fc8da95ea3943a264f4c57bcc1b6553bc832f4: Status 404 returned error can't find the container with id e35bceb916fe1e238994d84de9fc8da95ea3943a264f4c57bcc1b6553bc832f4 Sep 10 23:53:42.934994 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:42.934950 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" event={"ID":"a78a8a0e-80df-4ab3-b2af-eab9a3268037","Type":"ContainerStarted","Data":"a332d956e2c0e45acc07b4b692f48c96a28e210e45cde8ad206b6d7b2d64416e"} Sep 10 23:53:42.935480 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:42.935109 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" Sep 10 23:53:42.935480 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:42.935146 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" event={"ID":"a78a8a0e-80df-4ab3-b2af-eab9a3268037","Type":"ContainerStarted","Data":"e35bceb916fe1e238994d84de9fc8da95ea3943a264f4c57bcc1b6553bc832f4"} Sep 10 23:53:42.936698 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:42.936672 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" event={"ID":"374ad816-6d4b-4199-bdaa-47a58d95f543","Type":"ContainerStarted","Data":"db7f98c38eea07a440034868bb0702ec920c1eadd92158012089031150580f93"} Sep 10 23:53:42.936798 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:42.936721 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" Sep 10 23:53:42.936798 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:42.936746 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" event={"ID":"374ad816-6d4b-4199-bdaa-47a58d95f543","Type":"ContainerStarted","Data":"f0864bfb96bf0508b35d2fb9d548eb23d23deff32533973d165902af34dbea94"} Sep 10 23:53:42.962985 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:42.962921 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" podStartSLOduration=1.962905753 podStartE2EDuration="1.962905753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:53:42.962129004 +0000 UTC m=+1030.264607021" watchObservedRunningTime="2026-09-10 23:53:42.962905753 +0000 UTC m=+1030.265383764" Sep 10 23:53:53.944239 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:53.944201 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" Sep 10 23:53:53.944723 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:53.944277 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" Sep 10 23:53:53.964629 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:53.964573 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" podStartSLOduration=12.96455427 podStartE2EDuration="12.96455427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:53:42.997342847 +0000 UTC m=+1030.299820856" watchObservedRunningTime="2026-09-10 23:53:53.96455427 +0000 UTC m=+1041.267032283" Sep 10 23:53:58.014654 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:58.014564 2583 generic.go:356] "Generic (PLEG): container finished" podID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerID="03dd36eaa0c46c6fadace282efcb1dc3ea413b608f0d2ce103b3770d87c21292" exitCode=0 Sep 10 23:53:58.015042 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:58.014641 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" event={"ID":"6d52fe77-353b-4e07-a7b1-79ee73baa3c6","Type":"ContainerDied","Data":"03dd36eaa0c46c6fadace282efcb1dc3ea413b608f0d2ce103b3770d87c21292"} Sep 10 23:53:58.015311 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:58.015046 2583 scope.go:117] "RemoveContainer" containerID="03dd36eaa0c46c6fadace282efcb1dc3ea413b608f0d2ce103b3770d87c21292" Sep 10 23:53:58.016152 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:58.016126 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:53:58.016227 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:58.016209 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:53:58.016263 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:58.016256 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:53:59.022645 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:59.022603 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" event={"ID":"6d52fe77-353b-4e07-a7b1-79ee73baa3c6","Type":"ContainerStarted","Data":"91c096190ae9ab81a787741856bc0e9290448dd1bd80296b6e4f6cd74325cff9"} Sep 10 23:53:59.474670 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:59.474630 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:53:59.474850 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:59.474829 2583 patch_prober.go:28] interesting pod/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 10 23:53:59.474912 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:53:59.474886 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" podUID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 10 23:54:00.474508 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:00.474469 2583 patch_prober.go:28] interesting pod/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 10 23:54:00.474905 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:00.474526 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" podUID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 10 23:54:01.474632 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:01.474597 2583 patch_prober.go:28] interesting pod/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 10 23:54:01.475014 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:01.474657 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" podUID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 10 23:54:02.474392 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:02.474345 2583 patch_prober.go:28] interesting pod/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 10 23:54:02.474578 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:02.474410 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" podUID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 10 23:54:02.493226 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:02.493184 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74"] Sep 10 23:54:03.475048 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:03.475008 2583 patch_prober.go:28] interesting pod/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 10 23:54:03.475252 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:03.475065 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" podUID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 10 23:54:04.480938 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:04.480897 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:54:04.481427 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:04.481266 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:54:04.482263 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:04.482236 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:54:08.437912 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.437879 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp"] Sep 10 23:54:08.441265 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.441245 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" Sep 10 23:54:08.443986 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.443971 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fe49c344-dockercfg-m9hv4\"" Sep 10 23:54:08.444389 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.444375 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fe49c344\"" Sep 10 23:54:08.452383 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.452360 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp"] Sep 10 23:54:08.458447 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.458427 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s"] Sep 10 23:54:08.461640 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.461625 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" Sep 10 23:54:08.483487 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.483465 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s"] Sep 10 23:54:08.509557 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.509531 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2k6\" (UniqueName: \"kubernetes.io/projected/655e3c13-45eb-4561-8d18-1e6611ca01ea-kube-api-access-5t2k6\") pod \"payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp\" (UID: \"655e3c13-45eb-4561-8d18-1e6611ca01ea\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" Sep 10 23:54:08.509663 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.509577 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655e3c13-45eb-4561-8d18-1e6611ca01ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp\" (UID: \"655e3c13-45eb-4561-8d18-1e6611ca01ea\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" Sep 10 23:54:08.509663 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.509638 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b395fc-be60-407f-8bd3-3910b63e31a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s\" (UID: \"93b395fc-be60-407f-8bd3-3910b63e31a5\") " pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" Sep 10 23:54:08.509741 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.509679 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7mqt\" (UniqueName: \"kubernetes.io/projected/93b395fc-be60-407f-8bd3-3910b63e31a5-kube-api-access-l7mqt\") pod \"payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s\" (UID: \"93b395fc-be60-407f-8bd3-3910b63e31a5\") " pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" Sep 10 23:54:08.610176 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.610140 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655e3c13-45eb-4561-8d18-1e6611ca01ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp\" (UID: \"655e3c13-45eb-4561-8d18-1e6611ca01ea\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" Sep 10 23:54:08.610338 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.610193 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b395fc-be60-407f-8bd3-3910b63e31a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s\" (UID: \"93b395fc-be60-407f-8bd3-3910b63e31a5\") " pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" Sep 10 23:54:08.610338 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.610314 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7mqt\" (UniqueName: \"kubernetes.io/projected/93b395fc-be60-407f-8bd3-3910b63e31a5-kube-api-access-l7mqt\") pod \"payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s\" (UID: \"93b395fc-be60-407f-8bd3-3910b63e31a5\") " pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" Sep 10 23:54:08.610498 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.610481 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2k6\" (UniqueName: \"kubernetes.io/projected/655e3c13-45eb-4561-8d18-1e6611ca01ea-kube-api-access-5t2k6\") pod \"payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp\" (UID: \"655e3c13-45eb-4561-8d18-1e6611ca01ea\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" Sep 10 23:54:08.610840 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.610820 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b395fc-be60-407f-8bd3-3910b63e31a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s\" (UID: \"93b395fc-be60-407f-8bd3-3910b63e31a5\") " pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" Sep 10 23:54:08.610951 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.610892 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655e3c13-45eb-4561-8d18-1e6611ca01ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp\" (UID: \"655e3c13-45eb-4561-8d18-1e6611ca01ea\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" Sep 10 23:54:08.619496 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.619469 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2k6\" (UniqueName: \"kubernetes.io/projected/655e3c13-45eb-4561-8d18-1e6611ca01ea-kube-api-access-5t2k6\") pod \"payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp\" (UID: \"655e3c13-45eb-4561-8d18-1e6611ca01ea\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" Sep 10 23:54:08.619686 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.619670 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7mqt\" (UniqueName: \"kubernetes.io/projected/93b395fc-be60-407f-8bd3-3910b63e31a5-kube-api-access-l7mqt\") pod \"payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s\" (UID: \"93b395fc-be60-407f-8bd3-3910b63e31a5\") " pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" Sep 10 23:54:08.752953 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.752866 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" Sep 10 23:54:08.772261 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:08.772234 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" Sep 10 23:54:09.059430 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:09.059342 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp"] Sep 10 23:54:09.062425 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:54:09.062401 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655e3c13_45eb_4561_8d18_1e6611ca01ea.slice/crio-511f906e29ae7c89e718e5526ce662e264184be0a0f2cc787d04196f4eb9478b WatchSource:0}: Error finding container 511f906e29ae7c89e718e5526ce662e264184be0a0f2cc787d04196f4eb9478b: Status 404 returned error can't find the container with id 511f906e29ae7c89e718e5526ce662e264184be0a0f2cc787d04196f4eb9478b Sep 10 23:54:09.081780 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:09.081749 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" event={"ID":"655e3c13-45eb-4561-8d18-1e6611ca01ea","Type":"ContainerStarted","Data":"511f906e29ae7c89e718e5526ce662e264184be0a0f2cc787d04196f4eb9478b"} Sep 10 23:54:09.091593 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:09.091570 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s"] Sep 10 23:54:09.095185 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:54:09.095152 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b395fc_be60_407f_8bd3_3910b63e31a5.slice/crio-0c603124507b5033c73fc4a178f102beefcbbfc39f70563a54aaa253136236da WatchSource:0}: Error finding container 0c603124507b5033c73fc4a178f102beefcbbfc39f70563a54aaa253136236da: Status 404 returned error can't find the container with id 0c603124507b5033c73fc4a178f102beefcbbfc39f70563a54aaa253136236da Sep 10 23:54:10.088809 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:10.088760 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" event={"ID":"655e3c13-45eb-4561-8d18-1e6611ca01ea","Type":"ContainerStarted","Data":"0227acbf3a0625cb07c3c0aee71747b0129a335b0dbd917faea64433a2689ab4"} Sep 10 23:54:10.089245 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:10.088834 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" Sep 10 23:54:10.090427 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:10.090406 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" event={"ID":"93b395fc-be60-407f-8bd3-3910b63e31a5","Type":"ContainerStarted","Data":"95e5c8e56ccd294d50e0740956c1a192fbabd3fd96d530ad00b12bc9cfd779a3"} Sep 10 23:54:10.090503 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:10.090456 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" Sep 10 23:54:10.090503 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:10.090473 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" event={"ID":"93b395fc-be60-407f-8bd3-3910b63e31a5","Type":"ContainerStarted","Data":"0c603124507b5033c73fc4a178f102beefcbbfc39f70563a54aaa253136236da"} Sep 10 23:54:10.107104 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:10.107035 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" podStartSLOduration=2.107016982 podStartE2EDuration="2.107016982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:10.106254547 +0000 UTC m=+1057.408732573" watchObservedRunningTime="2026-09-10 23:54:10.107016982 +0000 UTC m=+1057.409494996" Sep 10 23:54:10.123662 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:10.123614 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" podStartSLOduration=2.123599646 podStartE2EDuration="2.123599646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:10.122349051 +0000 UTC m=+1057.424827068" watchObservedRunningTime="2026-09-10 23:54:10.123599646 +0000 UTC m=+1057.426077657" Sep 10 23:54:17.512060 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.512019 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" podUID="2cb5c73b-a237-475e-92b1-6b7e04517619" containerName="main" containerID="cri-o://4817c4fd28b5d7a264906f4107970e16710bac95324198e3002ef47d17a9441b" gracePeriod=45 Sep 10 23:54:17.755344 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.755319 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:54:17.795662 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.795596 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb5c73b-a237-475e-92b1-6b7e04517619-tls-certs\") pod \"2cb5c73b-a237-475e-92b1-6b7e04517619\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " Sep 10 23:54:17.795662 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.795646 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-model-cache\") pod \"2cb5c73b-a237-475e-92b1-6b7e04517619\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " Sep 10 23:54:17.795838 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.795669 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-tmp-dir\") pod \"2cb5c73b-a237-475e-92b1-6b7e04517619\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " Sep 10 23:54:17.795838 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.795722 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-dshm\") pod \"2cb5c73b-a237-475e-92b1-6b7e04517619\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " Sep 10 23:54:17.795838 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.795770 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gtfw\" (UniqueName: \"kubernetes.io/projected/2cb5c73b-a237-475e-92b1-6b7e04517619-kube-api-access-7gtfw\") pod \"2cb5c73b-a237-475e-92b1-6b7e04517619\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " Sep 10 23:54:17.795838 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.795802 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-home\") pod \"2cb5c73b-a237-475e-92b1-6b7e04517619\" (UID: \"2cb5c73b-a237-475e-92b1-6b7e04517619\") " Sep 10 23:54:17.796054 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.795905 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cb5c73b-a237-475e-92b1-6b7e04517619" (UID: "2cb5c73b-a237-475e-92b1-6b7e04517619"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:17.796054 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.795969 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2cb5c73b-a237-475e-92b1-6b7e04517619" (UID: "2cb5c73b-a237-475e-92b1-6b7e04517619"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:17.796192 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.796153 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-home" (OuterVolumeSpecName: "home") pod "2cb5c73b-a237-475e-92b1-6b7e04517619" (UID: "2cb5c73b-a237-475e-92b1-6b7e04517619"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:17.796251 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.796205 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-model-cache\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:17.796251 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.796222 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-tmp-dir\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:17.797873 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.797835 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-dshm" (OuterVolumeSpecName: "dshm") pod "2cb5c73b-a237-475e-92b1-6b7e04517619" (UID: "2cb5c73b-a237-475e-92b1-6b7e04517619"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:17.797997 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.797974 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb5c73b-a237-475e-92b1-6b7e04517619-kube-api-access-7gtfw" (OuterVolumeSpecName: "kube-api-access-7gtfw") pod "2cb5c73b-a237-475e-92b1-6b7e04517619" (UID: "2cb5c73b-a237-475e-92b1-6b7e04517619"). InnerVolumeSpecName "kube-api-access-7gtfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:17.798120 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.798093 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb5c73b-a237-475e-92b1-6b7e04517619-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2cb5c73b-a237-475e-92b1-6b7e04517619" (UID: "2cb5c73b-a237-475e-92b1-6b7e04517619"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:54:17.897512 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.897484 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gtfw\" (UniqueName: \"kubernetes.io/projected/2cb5c73b-a237-475e-92b1-6b7e04517619-kube-api-access-7gtfw\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:17.897512 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.897507 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-home\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:17.897512 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.897517 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb5c73b-a237-475e-92b1-6b7e04517619-tls-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:17.897725 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:17.897526 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb5c73b-a237-475e-92b1-6b7e04517619-dshm\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:18.131825 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:18.131791 2583 generic.go:356] "Generic (PLEG): container finished" podID="2cb5c73b-a237-475e-92b1-6b7e04517619" containerID="4817c4fd28b5d7a264906f4107970e16710bac95324198e3002ef47d17a9441b" exitCode=0 Sep 10 23:54:18.132000 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:18.131836 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" Sep 10 23:54:18.132000 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:18.131875 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" event={"ID":"2cb5c73b-a237-475e-92b1-6b7e04517619","Type":"ContainerDied","Data":"4817c4fd28b5d7a264906f4107970e16710bac95324198e3002ef47d17a9441b"} Sep 10 23:54:18.132000 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:18.131923 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74" event={"ID":"2cb5c73b-a237-475e-92b1-6b7e04517619","Type":"ContainerDied","Data":"9e310ba56341cbf28301ff7e5836160f233abb5ddc6e570a65bc2ffb8049ba09"} Sep 10 23:54:18.132000 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:18.131948 2583 scope.go:117] "RemoveContainer" containerID="4817c4fd28b5d7a264906f4107970e16710bac95324198e3002ef47d17a9441b" Sep 10 23:54:18.140925 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:18.140908 2583 scope.go:117] "RemoveContainer" containerID="4817c4fd28b5d7a264906f4107970e16710bac95324198e3002ef47d17a9441b" Sep 10 23:54:18.141212 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:54:18.141192 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4817c4fd28b5d7a264906f4107970e16710bac95324198e3002ef47d17a9441b\": container with ID starting with 4817c4fd28b5d7a264906f4107970e16710bac95324198e3002ef47d17a9441b not found: ID does not exist" containerID="4817c4fd28b5d7a264906f4107970e16710bac95324198e3002ef47d17a9441b" Sep 10 23:54:18.141260 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:18.141221 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4817c4fd28b5d7a264906f4107970e16710bac95324198e3002ef47d17a9441b"} err="failed to get container status \"4817c4fd28b5d7a264906f4107970e16710bac95324198e3002ef47d17a9441b\": rpc error: code = NotFound desc = could not find container \"4817c4fd28b5d7a264906f4107970e16710bac95324198e3002ef47d17a9441b\": container with ID starting with 4817c4fd28b5d7a264906f4107970e16710bac95324198e3002ef47d17a9441b not found: ID does not exist" Sep 10 23:54:18.154621 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:18.154592 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74"] Sep 10 23:54:18.158323 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:18.158301 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-ca420b8d-kserve-575b9689f9-f5c74"] Sep 10 23:54:19.307909 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:19.307874 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cb5c73b-a237-475e-92b1-6b7e04517619" path="/var/lib/kubelet/pods/2cb5c73b-a237-475e-92b1-6b7e04517619/volumes" Sep 10 23:54:21.097806 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:21.097775 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" Sep 10 23:54:21.098218 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:21.097848 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" Sep 10 23:54:37.613716 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:37.613684 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm"] Sep 10 23:54:37.614090 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:37.614016 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cb5c73b-a237-475e-92b1-6b7e04517619" containerName="main" Sep 10 23:54:37.614090 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:37.614027 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cb5c73b-a237-475e-92b1-6b7e04517619" containerName="main" Sep 10 23:54:37.614167 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:37.614111 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cb5c73b-a237-475e-92b1-6b7e04517619" containerName="main" Sep 10 23:54:37.618424 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:37.618407 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm" Sep 10 23:54:37.627897 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:37.627878 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 10 23:54:37.628015 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:37.627978 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 23:54:37.638030 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:37.638008 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm"] Sep 10 23:54:37.767113 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:37.767049 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb759512-cd59-4f91-822e-bec49c99527d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm\" (UID: \"cb759512-cd59-4f91-822e-bec49c99527d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm" Sep 10 23:54:37.868468 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:37.868389 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb759512-cd59-4f91-822e-bec49c99527d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm\" (UID: \"cb759512-cd59-4f91-822e-bec49c99527d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm" Sep 10 23:54:37.869011 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:37.868991 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb759512-cd59-4f91-822e-bec49c99527d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm\" (UID: \"cb759512-cd59-4f91-822e-bec49c99527d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm" Sep 10 23:54:37.928273 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:37.928244 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm" Sep 10 23:54:38.206177 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:38.206119 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm"] Sep 10 23:54:38.208427 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:54:38.208395 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb759512_cd59_4f91_822e_bec49c99527d.slice/crio-60cd32121f74383fa76a0b566cd4379e5f6e5207076273c2b08abcb84a5ca723 WatchSource:0}: Error finding container 60cd32121f74383fa76a0b566cd4379e5f6e5207076273c2b08abcb84a5ca723: Status 404 returned error can't find the container with id 60cd32121f74383fa76a0b566cd4379e5f6e5207076273c2b08abcb84a5ca723 Sep 10 23:54:38.232854 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:38.232829 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm" event={"ID":"cb759512-cd59-4f91-822e-bec49c99527d","Type":"ContainerStarted","Data":"60cd32121f74383fa76a0b566cd4379e5f6e5207076273c2b08abcb84a5ca723"} Sep 10 23:54:38.834893 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:38.834794 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j"] Sep 10 23:54:38.839913 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:38.839884 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j" Sep 10 23:54:38.850786 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:38.850751 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j"] Sep 10 23:54:38.980974 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:38.980947 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a870d63a-978a-4f71-8a8a-a22df49b87eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j\" (UID: \"a870d63a-978a-4f71-8a8a-a22df49b87eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j" Sep 10 23:54:39.081950 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:39.081910 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a870d63a-978a-4f71-8a8a-a22df49b87eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j\" (UID: \"a870d63a-978a-4f71-8a8a-a22df49b87eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j" Sep 10 23:54:39.082552 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:39.082529 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a870d63a-978a-4f71-8a8a-a22df49b87eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j\" (UID: \"a870d63a-978a-4f71-8a8a-a22df49b87eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j" Sep 10 23:54:39.152235 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:39.152202 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j" Sep 10 23:54:39.239168 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:39.239131 2583 generic.go:356] "Generic (PLEG): container finished" podID="cb759512-cd59-4f91-822e-bec49c99527d" containerID="612672f25e6737fc78545c1d5a22adf4074fcacae29a1ecc1e54c274d6665f33" exitCode=0 Sep 10 23:54:39.239330 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:39.239246 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm" event={"ID":"cb759512-cd59-4f91-822e-bec49c99527d","Type":"ContainerDied","Data":"612672f25e6737fc78545c1d5a22adf4074fcacae29a1ecc1e54c274d6665f33"} Sep 10 23:54:39.445999 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:39.445922 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j"] Sep 10 23:54:39.448919 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:54:39.448883 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda870d63a_978a_4f71_8a8a_a22df49b87eb.slice/crio-9e32164ee941cb90347083e15ca12a7564a5c9cd6f5687d468b55acafaa5446b WatchSource:0}: Error finding container 9e32164ee941cb90347083e15ca12a7564a5c9cd6f5687d468b55acafaa5446b: Status 404 returned error can't find the container with id 9e32164ee941cb90347083e15ca12a7564a5c9cd6f5687d468b55acafaa5446b Sep 10 23:54:40.247026 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:40.246991 2583 generic.go:356] "Generic (PLEG): container finished" podID="a870d63a-978a-4f71-8a8a-a22df49b87eb" containerID="29d72ccca5b1187d41321851cf5913e44fac906dc7aa0124d5a999bbd3f590c5" exitCode=0 Sep 10 23:54:40.247404 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:40.247084 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j" event={"ID":"a870d63a-978a-4f71-8a8a-a22df49b87eb","Type":"ContainerDied","Data":"29d72ccca5b1187d41321851cf5913e44fac906dc7aa0124d5a999bbd3f590c5"} Sep 10 23:54:40.247404 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:40.247169 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j" event={"ID":"a870d63a-978a-4f71-8a8a-a22df49b87eb","Type":"ContainerStarted","Data":"9e32164ee941cb90347083e15ca12a7564a5c9cd6f5687d468b55acafaa5446b"} Sep 10 23:54:40.384127 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:40.384106 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm" Sep 10 23:54:40.492756 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:40.492653 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb759512-cd59-4f91-822e-bec49c99527d-maas-api-service-ca\") pod \"cb759512-cd59-4f91-822e-bec49c99527d\" (UID: \"cb759512-cd59-4f91-822e-bec49c99527d\") " Sep 10 23:54:40.493029 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:40.493005 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb759512-cd59-4f91-822e-bec49c99527d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb759512-cd59-4f91-822e-bec49c99527d" (UID: "cb759512-cd59-4f91-822e-bec49c99527d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:54:40.593839 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:40.593812 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb759512-cd59-4f91-822e-bec49c99527d-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:41.253749 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:41.253717 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm" Sep 10 23:54:41.254163 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:41.253749 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm" event={"ID":"cb759512-cd59-4f91-822e-bec49c99527d","Type":"ContainerDied","Data":"60cd32121f74383fa76a0b566cd4379e5f6e5207076273c2b08abcb84a5ca723"} Sep 10 23:54:41.254163 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:41.253800 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60cd32121f74383fa76a0b566cd4379e5f6e5207076273c2b08abcb84a5ca723" Sep 10 23:54:41.381315 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:41.381294 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j" Sep 10 23:54:41.502850 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:41.502815 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a870d63a-978a-4f71-8a8a-a22df49b87eb-maas-api-service-ca\") pod \"a870d63a-978a-4f71-8a8a-a22df49b87eb\" (UID: \"a870d63a-978a-4f71-8a8a-a22df49b87eb\") " Sep 10 23:54:41.503234 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:41.503213 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a870d63a-978a-4f71-8a8a-a22df49b87eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a870d63a-978a-4f71-8a8a-a22df49b87eb" (UID: "a870d63a-978a-4f71-8a8a-a22df49b87eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:54:41.604389 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:41.604363 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a870d63a-978a-4f71-8a8a-a22df49b87eb-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:42.260538 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:42.260507 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j" Sep 10 23:54:42.260992 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:42.260555 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j" event={"ID":"a870d63a-978a-4f71-8a8a-a22df49b87eb","Type":"ContainerDied","Data":"9e32164ee941cb90347083e15ca12a7564a5c9cd6f5687d468b55acafaa5446b"} Sep 10 23:54:42.260992 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:42.260602 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e32164ee941cb90347083e15ca12a7564a5c9cd6f5687d468b55acafaa5446b" Sep 10 23:54:48.462252 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.462219 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s"] Sep 10 23:54:48.462713 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.462431 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" podUID="93b395fc-be60-407f-8bd3-3910b63e31a5" containerName="payload-processing" containerID="cri-o://95e5c8e56ccd294d50e0740956c1a192fbabd3fd96d530ad00b12bc9cfd779a3" gracePeriod=30 Sep 10 23:54:48.475468 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.475432 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp"] Sep 10 23:54:48.475468 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.475672 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" podUID="655e3c13-45eb-4561-8d18-1e6611ca01ea" containerName="payload-pre-processing" containerID="cri-o://0227acbf3a0625cb07c3c0aee71747b0129a335b0dbd917faea64433a2689ab4" gracePeriod=30 Sep 10 23:54:48.685889 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.685857 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm"] Sep 10 23:54:48.689570 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.689538 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe49c344-6hsjm"] Sep 10 23:54:48.738448 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.738427 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" Sep 10 23:54:48.741082 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.741053 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" Sep 10 23:54:48.863286 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.863242 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b395fc-be60-407f-8bd3-3910b63e31a5-plugins-config-volume\") pod \"93b395fc-be60-407f-8bd3-3910b63e31a5\" (UID: \"93b395fc-be60-407f-8bd3-3910b63e31a5\") " Sep 10 23:54:48.863286 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.863292 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655e3c13-45eb-4561-8d18-1e6611ca01ea-plugins-config-volume\") pod \"655e3c13-45eb-4561-8d18-1e6611ca01ea\" (UID: \"655e3c13-45eb-4561-8d18-1e6611ca01ea\") " Sep 10 23:54:48.863553 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.863340 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t2k6\" (UniqueName: \"kubernetes.io/projected/655e3c13-45eb-4561-8d18-1e6611ca01ea-kube-api-access-5t2k6\") pod \"655e3c13-45eb-4561-8d18-1e6611ca01ea\" (UID: \"655e3c13-45eb-4561-8d18-1e6611ca01ea\") " Sep 10 23:54:48.863553 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.863430 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7mqt\" (UniqueName: \"kubernetes.io/projected/93b395fc-be60-407f-8bd3-3910b63e31a5-kube-api-access-l7mqt\") pod \"93b395fc-be60-407f-8bd3-3910b63e31a5\" (UID: \"93b395fc-be60-407f-8bd3-3910b63e31a5\") " Sep 10 23:54:48.863669 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.863596 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b395fc-be60-407f-8bd3-3910b63e31a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "93b395fc-be60-407f-8bd3-3910b63e31a5" (UID: "93b395fc-be60-407f-8bd3-3910b63e31a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:54:48.863737 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.863717 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b395fc-be60-407f-8bd3-3910b63e31a5-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:48.863793 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.863751 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655e3c13-45eb-4561-8d18-1e6611ca01ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "655e3c13-45eb-4561-8d18-1e6611ca01ea" (UID: "655e3c13-45eb-4561-8d18-1e6611ca01ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:54:48.865549 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.865515 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655e3c13-45eb-4561-8d18-1e6611ca01ea-kube-api-access-5t2k6" (OuterVolumeSpecName: "kube-api-access-5t2k6") pod "655e3c13-45eb-4561-8d18-1e6611ca01ea" (UID: "655e3c13-45eb-4561-8d18-1e6611ca01ea"). InnerVolumeSpecName "kube-api-access-5t2k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:48.865549 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.865526 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b395fc-be60-407f-8bd3-3910b63e31a5-kube-api-access-l7mqt" (OuterVolumeSpecName: "kube-api-access-l7mqt") pod "93b395fc-be60-407f-8bd3-3910b63e31a5" (UID: "93b395fc-be60-407f-8bd3-3910b63e31a5"). InnerVolumeSpecName "kube-api-access-l7mqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:48.964194 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.964164 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655e3c13-45eb-4561-8d18-1e6611ca01ea-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:48.964194 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.964189 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t2k6\" (UniqueName: \"kubernetes.io/projected/655e3c13-45eb-4561-8d18-1e6611ca01ea-kube-api-access-5t2k6\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:48.964194 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:48.964198 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7mqt\" (UniqueName: \"kubernetes.io/projected/93b395fc-be60-407f-8bd3-3910b63e31a5-kube-api-access-l7mqt\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:49.296146 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.296113 2583 generic.go:356] "Generic (PLEG): container finished" podID="655e3c13-45eb-4561-8d18-1e6611ca01ea" containerID="0227acbf3a0625cb07c3c0aee71747b0129a335b0dbd917faea64433a2689ab4" exitCode=0 Sep 10 23:54:49.296337 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.296160 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" Sep 10 23:54:49.296337 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.296201 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" event={"ID":"655e3c13-45eb-4561-8d18-1e6611ca01ea","Type":"ContainerDied","Data":"0227acbf3a0625cb07c3c0aee71747b0129a335b0dbd917faea64433a2689ab4"} Sep 10 23:54:49.296337 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.296251 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp" event={"ID":"655e3c13-45eb-4561-8d18-1e6611ca01ea","Type":"ContainerDied","Data":"511f906e29ae7c89e718e5526ce662e264184be0a0f2cc787d04196f4eb9478b"} Sep 10 23:54:49.296337 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.296279 2583 scope.go:117] "RemoveContainer" containerID="0227acbf3a0625cb07c3c0aee71747b0129a335b0dbd917faea64433a2689ab4" Sep 10 23:54:49.297652 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.297634 2583 generic.go:356] "Generic (PLEG): container finished" podID="93b395fc-be60-407f-8bd3-3910b63e31a5" containerID="95e5c8e56ccd294d50e0740956c1a192fbabd3fd96d530ad00b12bc9cfd779a3" exitCode=0 Sep 10 23:54:49.297764 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.297661 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" event={"ID":"93b395fc-be60-407f-8bd3-3910b63e31a5","Type":"ContainerDied","Data":"95e5c8e56ccd294d50e0740956c1a192fbabd3fd96d530ad00b12bc9cfd779a3"} Sep 10 23:54:49.297764 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.297678 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" Sep 10 23:54:49.297764 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.297690 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s" event={"ID":"93b395fc-be60-407f-8bd3-3910b63e31a5","Type":"ContainerDied","Data":"0c603124507b5033c73fc4a178f102beefcbbfc39f70563a54aaa253136236da"} Sep 10 23:54:49.306121 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.306003 2583 scope.go:117] "RemoveContainer" containerID="0227acbf3a0625cb07c3c0aee71747b0129a335b0dbd917faea64433a2689ab4" Sep 10 23:54:49.306478 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:54:49.306447 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0227acbf3a0625cb07c3c0aee71747b0129a335b0dbd917faea64433a2689ab4\": container with ID starting with 0227acbf3a0625cb07c3c0aee71747b0129a335b0dbd917faea64433a2689ab4 not found: ID does not exist" containerID="0227acbf3a0625cb07c3c0aee71747b0129a335b0dbd917faea64433a2689ab4" Sep 10 23:54:49.306596 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.306488 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0227acbf3a0625cb07c3c0aee71747b0129a335b0dbd917faea64433a2689ab4"} err="failed to get container status \"0227acbf3a0625cb07c3c0aee71747b0129a335b0dbd917faea64433a2689ab4\": rpc error: code = NotFound desc = could not find container \"0227acbf3a0625cb07c3c0aee71747b0129a335b0dbd917faea64433a2689ab4\": container with ID starting with 0227acbf3a0625cb07c3c0aee71747b0129a335b0dbd917faea64433a2689ab4 not found: ID does not exist" Sep 10 23:54:49.306596 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.306514 2583 scope.go:117] "RemoveContainer" containerID="95e5c8e56ccd294d50e0740956c1a192fbabd3fd96d530ad00b12bc9cfd779a3" Sep 10 23:54:49.309518 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.309496 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb759512-cd59-4f91-822e-bec49c99527d" path="/var/lib/kubelet/pods/cb759512-cd59-4f91-822e-bec49c99527d/volumes" Sep 10 23:54:49.315754 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.315738 2583 scope.go:117] "RemoveContainer" containerID="95e5c8e56ccd294d50e0740956c1a192fbabd3fd96d530ad00b12bc9cfd779a3" Sep 10 23:54:49.316024 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:54:49.316006 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e5c8e56ccd294d50e0740956c1a192fbabd3fd96d530ad00b12bc9cfd779a3\": container with ID starting with 95e5c8e56ccd294d50e0740956c1a192fbabd3fd96d530ad00b12bc9cfd779a3 not found: ID does not exist" containerID="95e5c8e56ccd294d50e0740956c1a192fbabd3fd96d530ad00b12bc9cfd779a3" Sep 10 23:54:49.316087 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.316031 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e5c8e56ccd294d50e0740956c1a192fbabd3fd96d530ad00b12bc9cfd779a3"} err="failed to get container status \"95e5c8e56ccd294d50e0740956c1a192fbabd3fd96d530ad00b12bc9cfd779a3\": rpc error: code = NotFound desc = could not find container \"95e5c8e56ccd294d50e0740956c1a192fbabd3fd96d530ad00b12bc9cfd779a3\": container with ID starting with 95e5c8e56ccd294d50e0740956c1a192fbabd3fd96d530ad00b12bc9cfd779a3 not found: ID does not exist" Sep 10 23:54:49.324572 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.324551 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s"] Sep 10 23:54:49.329104 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.329084 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fe49c344-85857c9b69-twg8s"] Sep 10 23:54:49.342960 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.342937 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp"] Sep 10 23:54:49.349236 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:49.349216 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fe49c344-86bf96d54-tbjzp"] Sep 10 23:54:50.726680 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:50.726644 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6"] Sep 10 23:54:50.727123 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:50.726911 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" podUID="a78a8a0e-80df-4ab3-b2af-eab9a3268037" containerName="payload-processing" containerID="cri-o://a332d956e2c0e45acc07b4b692f48c96a28e210e45cde8ad206b6d7b2d64416e" gracePeriod=30 Sep 10 23:54:50.740469 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:50.740441 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n"] Sep 10 23:54:50.740692 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:50.740668 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" podUID="374ad816-6d4b-4199-bdaa-47a58d95f543" containerName="payload-pre-processing" containerID="cri-o://db7f98c38eea07a440034868bb0702ec920c1eadd92158012089031150580f93" gracePeriod=30 Sep 10 23:54:50.936033 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:50.936001 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j"] Sep 10 23:54:50.942065 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:50.942036 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7bb7cb-flt5j"] Sep 10 23:54:51.001897 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.001873 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" Sep 10 23:54:51.021404 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.021384 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" Sep 10 23:54:51.184440 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.184404 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj54v\" (UniqueName: \"kubernetes.io/projected/374ad816-6d4b-4199-bdaa-47a58d95f543-kube-api-access-gj54v\") pod \"374ad816-6d4b-4199-bdaa-47a58d95f543\" (UID: \"374ad816-6d4b-4199-bdaa-47a58d95f543\") " Sep 10 23:54:51.184640 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.184482 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/374ad816-6d4b-4199-bdaa-47a58d95f543-plugins-config-volume\") pod \"374ad816-6d4b-4199-bdaa-47a58d95f543\" (UID: \"374ad816-6d4b-4199-bdaa-47a58d95f543\") " Sep 10 23:54:51.184640 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.184577 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwcph\" (UniqueName: \"kubernetes.io/projected/a78a8a0e-80df-4ab3-b2af-eab9a3268037-kube-api-access-hwcph\") pod \"a78a8a0e-80df-4ab3-b2af-eab9a3268037\" (UID: \"a78a8a0e-80df-4ab3-b2af-eab9a3268037\") " Sep 10 23:54:51.184640 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.184606 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78a8a0e-80df-4ab3-b2af-eab9a3268037-plugins-config-volume\") pod \"a78a8a0e-80df-4ab3-b2af-eab9a3268037\" (UID: \"a78a8a0e-80df-4ab3-b2af-eab9a3268037\") " Sep 10 23:54:51.184921 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.184833 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/374ad816-6d4b-4199-bdaa-47a58d95f543-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "374ad816-6d4b-4199-bdaa-47a58d95f543" (UID: "374ad816-6d4b-4199-bdaa-47a58d95f543"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:54:51.185003 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.184983 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78a8a0e-80df-4ab3-b2af-eab9a3268037-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a78a8a0e-80df-4ab3-b2af-eab9a3268037" (UID: "a78a8a0e-80df-4ab3-b2af-eab9a3268037"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:54:51.186528 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.186501 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78a8a0e-80df-4ab3-b2af-eab9a3268037-kube-api-access-hwcph" (OuterVolumeSpecName: "kube-api-access-hwcph") pod "a78a8a0e-80df-4ab3-b2af-eab9a3268037" (UID: "a78a8a0e-80df-4ab3-b2af-eab9a3268037"). InnerVolumeSpecName "kube-api-access-hwcph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:51.186584 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.186533 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374ad816-6d4b-4199-bdaa-47a58d95f543-kube-api-access-gj54v" (OuterVolumeSpecName: "kube-api-access-gj54v") pod "374ad816-6d4b-4199-bdaa-47a58d95f543" (UID: "374ad816-6d4b-4199-bdaa-47a58d95f543"). InnerVolumeSpecName "kube-api-access-gj54v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:51.285956 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.285887 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj54v\" (UniqueName: \"kubernetes.io/projected/374ad816-6d4b-4199-bdaa-47a58d95f543-kube-api-access-gj54v\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:51.285956 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.285910 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/374ad816-6d4b-4199-bdaa-47a58d95f543-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:51.285956 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.285919 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwcph\" (UniqueName: \"kubernetes.io/projected/a78a8a0e-80df-4ab3-b2af-eab9a3268037-kube-api-access-hwcph\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:51.285956 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.285927 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a78a8a0e-80df-4ab3-b2af-eab9a3268037-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:54:51.308440 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.308402 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="655e3c13-45eb-4561-8d18-1e6611ca01ea" path="/var/lib/kubelet/pods/655e3c13-45eb-4561-8d18-1e6611ca01ea/volumes" Sep 10 23:54:51.308848 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.308831 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93b395fc-be60-407f-8bd3-3910b63e31a5" path="/var/lib/kubelet/pods/93b395fc-be60-407f-8bd3-3910b63e31a5/volumes" Sep 10 23:54:51.309160 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.309146 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a870d63a-978a-4f71-8a8a-a22df49b87eb" path="/var/lib/kubelet/pods/a870d63a-978a-4f71-8a8a-a22df49b87eb/volumes" Sep 10 23:54:51.312110 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.312057 2583 generic.go:356] "Generic (PLEG): container finished" podID="a78a8a0e-80df-4ab3-b2af-eab9a3268037" containerID="a332d956e2c0e45acc07b4b692f48c96a28e210e45cde8ad206b6d7b2d64416e" exitCode=0 Sep 10 23:54:51.312223 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.312127 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" Sep 10 23:54:51.312223 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.312142 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" event={"ID":"a78a8a0e-80df-4ab3-b2af-eab9a3268037","Type":"ContainerDied","Data":"a332d956e2c0e45acc07b4b692f48c96a28e210e45cde8ad206b6d7b2d64416e"} Sep 10 23:54:51.312223 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.312204 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6" event={"ID":"a78a8a0e-80df-4ab3-b2af-eab9a3268037","Type":"ContainerDied","Data":"e35bceb916fe1e238994d84de9fc8da95ea3943a264f4c57bcc1b6553bc832f4"} Sep 10 23:54:51.312407 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.312235 2583 scope.go:117] "RemoveContainer" containerID="a332d956e2c0e45acc07b4b692f48c96a28e210e45cde8ad206b6d7b2d64416e" Sep 10 23:54:51.313739 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.313711 2583 generic.go:356] "Generic (PLEG): container finished" podID="374ad816-6d4b-4199-bdaa-47a58d95f543" containerID="db7f98c38eea07a440034868bb0702ec920c1eadd92158012089031150580f93" exitCode=0 Sep 10 23:54:51.313864 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.313739 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" event={"ID":"374ad816-6d4b-4199-bdaa-47a58d95f543","Type":"ContainerDied","Data":"db7f98c38eea07a440034868bb0702ec920c1eadd92158012089031150580f93"} Sep 10 23:54:51.313864 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.313768 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" Sep 10 23:54:51.313864 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.313797 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n" event={"ID":"374ad816-6d4b-4199-bdaa-47a58d95f543","Type":"ContainerDied","Data":"f0864bfb96bf0508b35d2fb9d548eb23d23deff32533973d165902af34dbea94"} Sep 10 23:54:51.323755 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.323738 2583 scope.go:117] "RemoveContainer" containerID="a332d956e2c0e45acc07b4b692f48c96a28e210e45cde8ad206b6d7b2d64416e" Sep 10 23:54:51.324027 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:54:51.324002 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a332d956e2c0e45acc07b4b692f48c96a28e210e45cde8ad206b6d7b2d64416e\": container with ID starting with a332d956e2c0e45acc07b4b692f48c96a28e210e45cde8ad206b6d7b2d64416e not found: ID does not exist" containerID="a332d956e2c0e45acc07b4b692f48c96a28e210e45cde8ad206b6d7b2d64416e" Sep 10 23:54:51.324110 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.324033 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a332d956e2c0e45acc07b4b692f48c96a28e210e45cde8ad206b6d7b2d64416e"} err="failed to get container status \"a332d956e2c0e45acc07b4b692f48c96a28e210e45cde8ad206b6d7b2d64416e\": rpc error: code = NotFound desc = could not find container \"a332d956e2c0e45acc07b4b692f48c96a28e210e45cde8ad206b6d7b2d64416e\": container with ID starting with a332d956e2c0e45acc07b4b692f48c96a28e210e45cde8ad206b6d7b2d64416e not found: ID does not exist" Sep 10 23:54:51.324110 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.324053 2583 scope.go:117] "RemoveContainer" containerID="db7f98c38eea07a440034868bb0702ec920c1eadd92158012089031150580f93" Sep 10 23:54:51.331241 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.331224 2583 scope.go:117] "RemoveContainer" containerID="db7f98c38eea07a440034868bb0702ec920c1eadd92158012089031150580f93" Sep 10 23:54:51.331486 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:54:51.331471 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7f98c38eea07a440034868bb0702ec920c1eadd92158012089031150580f93\": container with ID starting with db7f98c38eea07a440034868bb0702ec920c1eadd92158012089031150580f93 not found: ID does not exist" containerID="db7f98c38eea07a440034868bb0702ec920c1eadd92158012089031150580f93" Sep 10 23:54:51.331546 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.331489 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7f98c38eea07a440034868bb0702ec920c1eadd92158012089031150580f93"} err="failed to get container status \"db7f98c38eea07a440034868bb0702ec920c1eadd92158012089031150580f93\": rpc error: code = NotFound desc = could not find container \"db7f98c38eea07a440034868bb0702ec920c1eadd92158012089031150580f93\": container with ID starting with db7f98c38eea07a440034868bb0702ec920c1eadd92158012089031150580f93 not found: ID does not exist" Sep 10 23:54:51.339312 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.339264 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n"] Sep 10 23:54:51.341840 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.341820 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7bb7cb-77887784c8-khk8n"] Sep 10 23:54:51.353382 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.353361 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6"] Sep 10 23:54:51.363120 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:51.363100 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7bb7cb-644d9d87cc-8wbf6"] Sep 10 23:54:53.307203 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:53.307173 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="374ad816-6d4b-4199-bdaa-47a58d95f543" path="/var/lib/kubelet/pods/374ad816-6d4b-4199-bdaa-47a58d95f543/volumes" Sep 10 23:54:53.307620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:53.307525 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a78a8a0e-80df-4ab3-b2af-eab9a3268037" path="/var/lib/kubelet/pods/a78a8a0e-80df-4ab3-b2af-eab9a3268037/volumes" Sep 10 23:54:57.392189 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:57.392152 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8"] Sep 10 23:54:57.392627 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:57.392412 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" podUID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerName="istio-proxy" containerID="cri-o://91c096190ae9ab81a787741856bc0e9290448dd1bd80296b6e4f6cd74325cff9" gracePeriod=30 Sep 10 23:54:58.146047 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146013 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw"] Sep 10 23:54:58.146477 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146459 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655e3c13-45eb-4561-8d18-1e6611ca01ea" containerName="payload-pre-processing" Sep 10 23:54:58.146477 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146478 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="655e3c13-45eb-4561-8d18-1e6611ca01ea" containerName="payload-pre-processing" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146488 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374ad816-6d4b-4199-bdaa-47a58d95f543" containerName="payload-pre-processing" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146494 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="374ad816-6d4b-4199-bdaa-47a58d95f543" containerName="payload-pre-processing" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146506 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b395fc-be60-407f-8bd3-3910b63e31a5" containerName="payload-processing" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146512 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b395fc-be60-407f-8bd3-3910b63e31a5" containerName="payload-processing" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146520 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a78a8a0e-80df-4ab3-b2af-eab9a3268037" containerName="payload-processing" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146525 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a78a8a0e-80df-4ab3-b2af-eab9a3268037" containerName="payload-processing" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146530 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb759512-cd59-4f91-822e-bec49c99527d" containerName="revoke-keys" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146536 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb759512-cd59-4f91-822e-bec49c99527d" containerName="revoke-keys" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146543 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a870d63a-978a-4f71-8a8a-a22df49b87eb" containerName="revoke-keys" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146550 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a870d63a-978a-4f71-8a8a-a22df49b87eb" containerName="revoke-keys" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146599 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a78a8a0e-80df-4ab3-b2af-eab9a3268037" containerName="payload-processing" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146607 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="655e3c13-45eb-4561-8d18-1e6611ca01ea" containerName="payload-pre-processing" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146614 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb759512-cd59-4f91-822e-bec49c99527d" containerName="revoke-keys" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146621 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="374ad816-6d4b-4199-bdaa-47a58d95f543" containerName="payload-pre-processing" Sep 10 23:54:58.146620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146628 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b395fc-be60-407f-8bd3-3910b63e31a5" containerName="payload-processing" Sep 10 23:54:58.147378 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.146634 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a870d63a-978a-4f71-8a8a-a22df49b87eb" containerName="revoke-keys" Sep 10 23:54:58.150988 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.150966 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.156139 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.156122 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-76b16f-openshift-default-dockercfg-t4jfh\"" Sep 10 23:54:58.163291 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.163270 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw"] Sep 10 23:54:58.247246 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.247204 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istiod-ca-cert\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.247543 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.247521 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-workload-certs\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.247711 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.247695 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-envoy\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.247858 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.247842 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-podinfo\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.248011 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.247996 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-credential-socket\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.248172 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.248157 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq7mz\" (UniqueName: \"kubernetes.io/projected/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-kube-api-access-fq7mz\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.248332 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.248316 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-workload-socket\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.248488 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.248465 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-data\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.248612 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.248499 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-token\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.349709 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.349668 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istiod-ca-cert\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.349709 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.349714 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-workload-certs\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.349950 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.349833 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-envoy\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.349950 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.349893 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-podinfo\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.349950 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.349945 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-credential-socket\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.350174 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.349982 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq7mz\" (UniqueName: \"kubernetes.io/projected/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-kube-api-access-fq7mz\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.350174 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.350036 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-workload-socket\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.350174 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.350061 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-data\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.350174 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.350105 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-token\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.350366 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.350170 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-workload-certs\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.350489 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.350466 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istiod-ca-cert\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.350655 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.350491 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-credential-socket\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.350743 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.350579 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-workload-socket\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.350817 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.350692 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-data\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.352969 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.352948 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-podinfo\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.353089 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.352984 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-envoy\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.358844 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.358814 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq7mz\" (UniqueName: \"kubernetes.io/projected/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-kube-api-access-fq7mz\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.358963 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.358816 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-token\") pod \"e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.464130 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.464010 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:58.741025 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.740854 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw"] Sep 10 23:54:58.744193 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:54:58.744160 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a05d0e_cf4f_46ae_8230_222f9b197d5c.slice/crio-90bbd2e4aea883fc1e69d1ff4ad97a46a2db39cfc34f3fe02acb36396f7060c7 WatchSource:0}: Error finding container 90bbd2e4aea883fc1e69d1ff4ad97a46a2db39cfc34f3fe02acb36396f7060c7: Status 404 returned error can't find the container with id 90bbd2e4aea883fc1e69d1ff4ad97a46a2db39cfc34f3fe02acb36396f7060c7 Sep 10 23:54:58.746615 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.746584 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:54:58.746708 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.746657 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:54:58.746708 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:58.746690 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:54:59.359365 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:59.359302 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" event={"ID":"e2a05d0e-cf4f-46ae-8230-222f9b197d5c","Type":"ContainerStarted","Data":"19dcfe60166e3d1d088ecb3a46c6b638e66c80a68b0ff31f4ee65940fa8d8a5d"} Sep 10 23:54:59.359565 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:59.359383 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" event={"ID":"e2a05d0e-cf4f-46ae-8230-222f9b197d5c","Type":"ContainerStarted","Data":"90bbd2e4aea883fc1e69d1ff4ad97a46a2db39cfc34f3fe02acb36396f7060c7"} Sep 10 23:54:59.386997 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:59.386947 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" podStartSLOduration=1.386931387 podStartE2EDuration="1.386931387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:59.384293065 +0000 UTC m=+1106.686771076" watchObservedRunningTime="2026-09-10 23:54:59.386931387 +0000 UTC m=+1106.689409399" Sep 10 23:54:59.464607 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:59.464577 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:54:59.469507 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:54:59.469482 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:55:00.363962 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:00.363932 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:55:00.364842 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:00.364826 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:55:02.645049 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.645019 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:55:02.687333 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.687265 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-token\") pod \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " Sep 10 23:55:02.687333 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.687295 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-workload-certs\") pod \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " Sep 10 23:55:02.687558 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.687366 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-envoy\") pod \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " Sep 10 23:55:02.687558 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.687409 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-data\") pod \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " Sep 10 23:55:02.687558 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.687455 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-podinfo\") pod \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " Sep 10 23:55:02.687558 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.687478 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-workload-socket\") pod \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " Sep 10 23:55:02.687558 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.687523 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vp2q\" (UniqueName: \"kubernetes.io/projected/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-kube-api-access-5vp2q\") pod \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " Sep 10 23:55:02.687558 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.687549 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-credential-socket\") pod \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " Sep 10 23:55:02.687875 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.687602 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istiod-ca-cert\") pod \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\" (UID: \"6d52fe77-353b-4e07-a7b1-79ee73baa3c6\") " Sep 10 23:55:02.687875 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.687751 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "6d52fe77-353b-4e07-a7b1-79ee73baa3c6" (UID: "6d52fe77-353b-4e07-a7b1-79ee73baa3c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:02.687985 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.687885 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6d52fe77-353b-4e07-a7b1-79ee73baa3c6" (UID: "6d52fe77-353b-4e07-a7b1-79ee73baa3c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:02.687985 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.687974 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6d52fe77-353b-4e07-a7b1-79ee73baa3c6" (UID: "6d52fe77-353b-4e07-a7b1-79ee73baa3c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:02.688156 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.688134 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6d52fe77-353b-4e07-a7b1-79ee73baa3c6" (UID: "6d52fe77-353b-4e07-a7b1-79ee73baa3c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:02.688235 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.688205 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6d52fe77-353b-4e07-a7b1-79ee73baa3c6" (UID: "6d52fe77-353b-4e07-a7b1-79ee73baa3c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:55:02.688307 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.688296 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:02.688370 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.688313 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:02.688370 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.688327 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:02.688370 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.688340 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:02.688370 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.688353 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:02.689533 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.689498 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "6d52fe77-353b-4e07-a7b1-79ee73baa3c6" (UID: "6d52fe77-353b-4e07-a7b1-79ee73baa3c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:02.689820 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.689791 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-kube-api-access-5vp2q" (OuterVolumeSpecName: "kube-api-access-5vp2q") pod "6d52fe77-353b-4e07-a7b1-79ee73baa3c6" (UID: "6d52fe77-353b-4e07-a7b1-79ee73baa3c6"). InnerVolumeSpecName "kube-api-access-5vp2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:02.689925 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.689907 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6d52fe77-353b-4e07-a7b1-79ee73baa3c6" (UID: "6d52fe77-353b-4e07-a7b1-79ee73baa3c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:55:02.690242 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.690228 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6d52fe77-353b-4e07-a7b1-79ee73baa3c6" (UID: "6d52fe77-353b-4e07-a7b1-79ee73baa3c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:02.789001 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.788972 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:02.789001 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.788995 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:02.789001 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.789004 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vp2q\" (UniqueName: \"kubernetes.io/projected/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-kube-api-access-5vp2q\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:02.789230 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:02.789013 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d52fe77-353b-4e07-a7b1-79ee73baa3c6-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:03.379571 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:03.379536 2583 generic.go:356] "Generic (PLEG): container finished" podID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerID="91c096190ae9ab81a787741856bc0e9290448dd1bd80296b6e4f6cd74325cff9" exitCode=0 Sep 10 23:55:03.379721 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:03.379580 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" Sep 10 23:55:03.379721 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:03.379616 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" event={"ID":"6d52fe77-353b-4e07-a7b1-79ee73baa3c6","Type":"ContainerDied","Data":"91c096190ae9ab81a787741856bc0e9290448dd1bd80296b6e4f6cd74325cff9"} Sep 10 23:55:03.379721 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:03.379666 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8" event={"ID":"6d52fe77-353b-4e07-a7b1-79ee73baa3c6","Type":"ContainerDied","Data":"d97b5eb2afa6a7cb5e7d9cdd931adc1403d3855a1b54e607137bf82592ad0248"} Sep 10 23:55:03.379721 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:03.379689 2583 scope.go:117] "RemoveContainer" containerID="91c096190ae9ab81a787741856bc0e9290448dd1bd80296b6e4f6cd74325cff9" Sep 10 23:55:03.388103 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:03.388064 2583 scope.go:117] "RemoveContainer" containerID="03dd36eaa0c46c6fadace282efcb1dc3ea413b608f0d2ce103b3770d87c21292" Sep 10 23:55:03.396129 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:03.396107 2583 scope.go:117] "RemoveContainer" containerID="91c096190ae9ab81a787741856bc0e9290448dd1bd80296b6e4f6cd74325cff9" Sep 10 23:55:03.396560 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:55:03.396534 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c096190ae9ab81a787741856bc0e9290448dd1bd80296b6e4f6cd74325cff9\": container with ID starting with 91c096190ae9ab81a787741856bc0e9290448dd1bd80296b6e4f6cd74325cff9 not found: ID does not exist" containerID="91c096190ae9ab81a787741856bc0e9290448dd1bd80296b6e4f6cd74325cff9" Sep 10 23:55:03.396560 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:03.396571 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c096190ae9ab81a787741856bc0e9290448dd1bd80296b6e4f6cd74325cff9"} err="failed to get container status \"91c096190ae9ab81a787741856bc0e9290448dd1bd80296b6e4f6cd74325cff9\": rpc error: code = NotFound desc = could not find container \"91c096190ae9ab81a787741856bc0e9290448dd1bd80296b6e4f6cd74325cff9\": container with ID starting with 91c096190ae9ab81a787741856bc0e9290448dd1bd80296b6e4f6cd74325cff9 not found: ID does not exist" Sep 10 23:55:03.396560 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:03.396593 2583 scope.go:117] "RemoveContainer" containerID="03dd36eaa0c46c6fadace282efcb1dc3ea413b608f0d2ce103b3770d87c21292" Sep 10 23:55:03.397244 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:55:03.397110 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03dd36eaa0c46c6fadace282efcb1dc3ea413b608f0d2ce103b3770d87c21292\": container with ID starting with 03dd36eaa0c46c6fadace282efcb1dc3ea413b608f0d2ce103b3770d87c21292 not found: ID does not exist" containerID="03dd36eaa0c46c6fadace282efcb1dc3ea413b608f0d2ce103b3770d87c21292" Sep 10 23:55:03.397244 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:03.397142 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03dd36eaa0c46c6fadace282efcb1dc3ea413b608f0d2ce103b3770d87c21292"} err="failed to get container status \"03dd36eaa0c46c6fadace282efcb1dc3ea413b608f0d2ce103b3770d87c21292\": rpc error: code = NotFound desc = could not find container \"03dd36eaa0c46c6fadace282efcb1dc3ea413b608f0d2ce103b3770d87c21292\": container with ID starting with 03dd36eaa0c46c6fadace282efcb1dc3ea413b608f0d2ce103b3770d87c21292 not found: ID does not exist" Sep 10 23:55:03.398667 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:03.398648 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8"] Sep 10 23:55:03.402281 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:03.402259 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-7bb7cb-openshift-default-7c8fdb9fd8-sr2t8"] Sep 10 23:55:05.306908 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:05.306875 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" path="/var/lib/kubelet/pods/6d52fe77-353b-4e07-a7b1-79ee73baa3c6/volumes" Sep 10 23:55:09.265216 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.265180 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs"] Sep 10 23:55:09.265613 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.265492 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerName="istio-proxy" Sep 10 23:55:09.265613 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.265502 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerName="istio-proxy" Sep 10 23:55:09.265613 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.265512 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerName="istio-proxy" Sep 10 23:55:09.265613 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.265517 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerName="istio-proxy" Sep 10 23:55:09.265613 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.265567 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerName="istio-proxy" Sep 10 23:55:09.265613 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.265575 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d52fe77-353b-4e07-a7b1-79ee73baa3c6" containerName="istio-proxy" Sep 10 23:55:09.269973 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.269948 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.272364 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.272341 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-396319-openshift-default-dockercfg-cnngb\"" Sep 10 23:55:09.280744 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.280712 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs"] Sep 10 23:55:09.344217 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.344179 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-token\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.344392 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.344219 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrn7\" (UniqueName: \"kubernetes.io/projected/71c463d7-e804-4097-a2e3-c6a67d900db4-kube-api-access-pmrn7\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.344392 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.344289 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-podinfo\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.344392 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.344334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-envoy\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.344392 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.344354 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c463d7-e804-4097-a2e3-c6a67d900db4-istiod-ca-cert\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.344583 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.344441 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-workload-socket\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.344583 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.344465 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-workload-certs\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.344583 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.344503 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-data\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.344583 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.344538 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-credential-socket\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.445685 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.445650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-workload-socket\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.445685 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.445685 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-workload-certs\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.445895 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.445709 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-data\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.445895 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.445744 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-credential-socket\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.445895 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.445805 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-token\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.445895 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.445826 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrn7\" (UniqueName: \"kubernetes.io/projected/71c463d7-e804-4097-a2e3-c6a67d900db4-kube-api-access-pmrn7\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.445895 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.445874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-podinfo\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.446189 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.445902 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-envoy\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.446189 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.445925 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c463d7-e804-4097-a2e3-c6a67d900db4-istiod-ca-cert\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.446189 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.446107 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-workload-certs\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.446189 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.446138 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-data\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.446189 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.446185 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-workload-socket\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.446373 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.446231 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-credential-socket\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.446719 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.446700 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c463d7-e804-4097-a2e3-c6a67d900db4-istiod-ca-cert\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.448259 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.448242 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-envoy\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.448564 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.448545 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-podinfo\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.464481 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.464461 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-token\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.464676 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.464656 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrn7\" (UniqueName: \"kubernetes.io/projected/71c463d7-e804-4097-a2e3-c6a67d900db4-kube-api-access-pmrn7\") pod \"e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.582840 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.582735 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:09.872616 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.872589 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs"] Sep 10 23:55:09.873940 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:55:09.873912 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c463d7_e804_4097_a2e3_c6a67d900db4.slice/crio-28240618e9d718c466041ce0fdb905d70e5637ce41de250ce4407c03e586a3d8 WatchSource:0}: Error finding container 28240618e9d718c466041ce0fdb905d70e5637ce41de250ce4407c03e586a3d8: Status 404 returned error can't find the container with id 28240618e9d718c466041ce0fdb905d70e5637ce41de250ce4407c03e586a3d8 Sep 10 23:55:09.875978 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.875945 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:55:09.876047 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.876010 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:55:09.876047 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:09.876040 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:55:10.415418 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:10.415380 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" event={"ID":"71c463d7-e804-4097-a2e3-c6a67d900db4","Type":"ContainerStarted","Data":"321a76ad05daec47fa528cfe1097c221125366d48abfd4326b495cca059a4304"} Sep 10 23:55:10.415797 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:10.415459 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" event={"ID":"71c463d7-e804-4097-a2e3-c6a67d900db4","Type":"ContainerStarted","Data":"28240618e9d718c466041ce0fdb905d70e5637ce41de250ce4407c03e586a3d8"} Sep 10 23:55:10.437978 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:10.437929 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" podStartSLOduration=1.437913998 podStartE2EDuration="1.437913998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:10.43622688 +0000 UTC m=+1117.738704892" watchObservedRunningTime="2026-09-10 23:55:10.437913998 +0000 UTC m=+1117.740392007" Sep 10 23:55:10.583404 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:10.583370 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:10.587921 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:10.587899 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:11.420016 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:11.419987 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:11.421015 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:11.420996 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:14.766728 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:14.766685 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559"] Sep 10 23:55:14.770377 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:14.770359 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" Sep 10 23:55:14.772940 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:14.772912 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-396319\"" Sep 10 23:55:14.780926 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:14.780906 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559"] Sep 10 23:55:14.896541 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:14.896502 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4kbd\" (UniqueName: \"kubernetes.io/projected/c941c0e9-d399-4756-8dbc-4246de2604d0-kube-api-access-l4kbd\") pod \"payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559\" (UID: \"c941c0e9-d399-4756-8dbc-4246de2604d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" Sep 10 23:55:14.896876 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:14.896842 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c941c0e9-d399-4756-8dbc-4246de2604d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559\" (UID: \"c941c0e9-d399-4756-8dbc-4246de2604d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" Sep 10 23:55:14.998064 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:14.998030 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4kbd\" (UniqueName: \"kubernetes.io/projected/c941c0e9-d399-4756-8dbc-4246de2604d0-kube-api-access-l4kbd\") pod \"payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559\" (UID: \"c941c0e9-d399-4756-8dbc-4246de2604d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" Sep 10 23:55:14.998064 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:14.998091 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c941c0e9-d399-4756-8dbc-4246de2604d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559\" (UID: \"c941c0e9-d399-4756-8dbc-4246de2604d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" Sep 10 23:55:14.998735 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:14.998716 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c941c0e9-d399-4756-8dbc-4246de2604d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559\" (UID: \"c941c0e9-d399-4756-8dbc-4246de2604d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" Sep 10 23:55:15.007029 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:15.007007 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4kbd\" (UniqueName: \"kubernetes.io/projected/c941c0e9-d399-4756-8dbc-4246de2604d0-kube-api-access-l4kbd\") pod \"payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559\" (UID: \"c941c0e9-d399-4756-8dbc-4246de2604d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" Sep 10 23:55:15.081122 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:15.081033 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" Sep 10 23:55:15.352329 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:15.352303 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559"] Sep 10 23:55:15.353713 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:55:15.353685 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc941c0e9_d399_4756_8dbc_4246de2604d0.slice/crio-1f7c89c0003c94e9473d227d73b13187ad0daa5bac96924041bd18ccf0921006 WatchSource:0}: Error finding container 1f7c89c0003c94e9473d227d73b13187ad0daa5bac96924041bd18ccf0921006: Status 404 returned error can't find the container with id 1f7c89c0003c94e9473d227d73b13187ad0daa5bac96924041bd18ccf0921006 Sep 10 23:55:15.445524 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:15.445483 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" event={"ID":"c941c0e9-d399-4756-8dbc-4246de2604d0","Type":"ContainerStarted","Data":"fa3b3587212dd36bcef7eef867dbec58ae944cfeb7a6e2bda2acb92dfafb079a"} Sep 10 23:55:15.445673 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:15.445588 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" Sep 10 23:55:15.445673 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:15.445613 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" event={"ID":"c941c0e9-d399-4756-8dbc-4246de2604d0","Type":"ContainerStarted","Data":"1f7c89c0003c94e9473d227d73b13187ad0daa5bac96924041bd18ccf0921006"} Sep 10 23:55:15.467739 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:15.467695 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" podStartSLOduration=1.467681227 podStartE2EDuration="1.467681227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:15.465419723 +0000 UTC m=+1122.767897733" watchObservedRunningTime="2026-09-10 23:55:15.467681227 +0000 UTC m=+1122.770159238" Sep 10 23:55:26.452738 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:26.452706 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" Sep 10 23:55:30.526727 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:30.526694 2583 generic.go:356] "Generic (PLEG): container finished" podID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerID="321a76ad05daec47fa528cfe1097c221125366d48abfd4326b495cca059a4304" exitCode=0 Sep 10 23:55:30.527212 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:30.526767 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" event={"ID":"71c463d7-e804-4097-a2e3-c6a67d900db4","Type":"ContainerDied","Data":"321a76ad05daec47fa528cfe1097c221125366d48abfd4326b495cca059a4304"} Sep 10 23:55:30.527212 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:30.527184 2583 scope.go:117] "RemoveContainer" containerID="321a76ad05daec47fa528cfe1097c221125366d48abfd4326b495cca059a4304" Sep 10 23:55:30.528271 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:30.528247 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:55:30.528330 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:30.528313 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:55:30.528368 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:30.528348 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:55:31.534368 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:31.534332 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" event={"ID":"71c463d7-e804-4097-a2e3-c6a67d900db4","Type":"ContainerStarted","Data":"bc78727d178bb690b943c998b89c7f0f5fd18f8d53ef9523c27067669357d237"} Sep 10 23:55:31.583135 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:31.583108 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:31.583342 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:31.583316 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 23:55:31.583444 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:31.583383 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 23:55:32.583369 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:32.583328 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 23:55:32.583828 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:32.583467 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 23:55:33.583399 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:33.583312 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 23:55:33.583399 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:33.583365 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 23:55:34.356147 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:34.355278 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw"] Sep 10 23:55:34.356147 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:34.355517 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" podUID="e2a05d0e-cf4f-46ae-8230-222f9b197d5c" containerName="istio-proxy" containerID="cri-o://19dcfe60166e3d1d088ecb3a46c6b638e66c80a68b0ff31f4ee65940fa8d8a5d" gracePeriod=30 Sep 10 23:55:34.583621 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:34.583579 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 23:55:34.583994 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:34.583641 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 23:55:35.463324 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:35.463264 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c"] Sep 10 23:55:35.468257 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:35.468234 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c" Sep 10 23:55:35.470595 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:35.470575 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 23:55:35.471230 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:35.471213 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 10 23:55:35.475939 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:35.475919 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c"] Sep 10 23:55:35.583350 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:35.583310 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 23:55:35.583543 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:35.583404 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 23:55:35.588280 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:35.588255 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd1245a-79a4-4ead-8f39-99c2450eaa8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c\" (UID: \"ebd1245a-79a4-4ead-8f39-99c2450eaa8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c" Sep 10 23:55:35.689199 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:35.689165 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd1245a-79a4-4ead-8f39-99c2450eaa8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c\" (UID: \"ebd1245a-79a4-4ead-8f39-99c2450eaa8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c" Sep 10 23:55:35.689709 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:35.689690 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd1245a-79a4-4ead-8f39-99c2450eaa8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c\" (UID: \"ebd1245a-79a4-4ead-8f39-99c2450eaa8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c" Sep 10 23:55:35.779223 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:35.779148 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c" Sep 10 23:55:36.071616 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:36.071590 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c"] Sep 10 23:55:36.073718 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:55:36.073689 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd1245a_79a4_4ead_8f39_99c2450eaa8b.slice/crio-eb6a6aeb41f622577c4973d1624f83e785590e62fb33849b5ddf05ef8162a281 WatchSource:0}: Error finding container eb6a6aeb41f622577c4973d1624f83e785590e62fb33849b5ddf05ef8162a281: Status 404 returned error can't find the container with id eb6a6aeb41f622577c4973d1624f83e785590e62fb33849b5ddf05ef8162a281 Sep 10 23:55:36.566622 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:36.566588 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c" event={"ID":"ebd1245a-79a4-4ead-8f39-99c2450eaa8b","Type":"ContainerStarted","Data":"c19c0dbae5e3be9c4e32eaa1d663258dedbad48a310a940472a218dfe5f64c33"} Sep 10 23:55:36.566825 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:36.566636 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c" event={"ID":"ebd1245a-79a4-4ead-8f39-99c2450eaa8b","Type":"ContainerStarted","Data":"eb6a6aeb41f622577c4973d1624f83e785590e62fb33849b5ddf05ef8162a281"} Sep 10 23:55:36.582167 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:36.582117 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c" podStartSLOduration=1.582101678 podStartE2EDuration="1.582101678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:36.580656842 +0000 UTC m=+1143.883134852" watchObservedRunningTime="2026-09-10 23:55:36.582101678 +0000 UTC m=+1143.884579688" Sep 10 23:55:36.583582 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:36.583562 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 23:55:36.583644 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:36.583605 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 23:55:37.572298 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:37.572266 2583 generic.go:356] "Generic (PLEG): container finished" podID="ebd1245a-79a4-4ead-8f39-99c2450eaa8b" containerID="c19c0dbae5e3be9c4e32eaa1d663258dedbad48a310a940472a218dfe5f64c33" exitCode=0 Sep 10 23:55:37.572652 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:37.572356 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c" event={"ID":"ebd1245a-79a4-4ead-8f39-99c2450eaa8b","Type":"ContainerDied","Data":"c19c0dbae5e3be9c4e32eaa1d663258dedbad48a310a940472a218dfe5f64c33"} Sep 10 23:55:37.587157 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:37.587136 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:37.587411 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:37.587395 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:37.587909 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:37.587894 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 10 23:55:38.703826 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:38.703799 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c" Sep 10 23:55:38.817959 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:38.817930 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd1245a-79a4-4ead-8f39-99c2450eaa8b-maas-api-service-ca\") pod \"ebd1245a-79a4-4ead-8f39-99c2450eaa8b\" (UID: \"ebd1245a-79a4-4ead-8f39-99c2450eaa8b\") " Sep 10 23:55:38.818297 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:38.818275 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd1245a-79a4-4ead-8f39-99c2450eaa8b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebd1245a-79a4-4ead-8f39-99c2450eaa8b" (UID: "ebd1245a-79a4-4ead-8f39-99c2450eaa8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:55:38.919494 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:38.919459 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd1245a-79a4-4ead-8f39-99c2450eaa8b-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:39.585548 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.585454 2583 generic.go:356] "Generic (PLEG): container finished" podID="e2a05d0e-cf4f-46ae-8230-222f9b197d5c" containerID="19dcfe60166e3d1d088ecb3a46c6b638e66c80a68b0ff31f4ee65940fa8d8a5d" exitCode=0 Sep 10 23:55:39.585714 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.585538 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" event={"ID":"e2a05d0e-cf4f-46ae-8230-222f9b197d5c","Type":"ContainerDied","Data":"19dcfe60166e3d1d088ecb3a46c6b638e66c80a68b0ff31f4ee65940fa8d8a5d"} Sep 10 23:55:39.587162 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.587133 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c" event={"ID":"ebd1245a-79a4-4ead-8f39-99c2450eaa8b","Type":"ContainerDied","Data":"eb6a6aeb41f622577c4973d1624f83e785590e62fb33849b5ddf05ef8162a281"} Sep 10 23:55:39.587162 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.587159 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c" Sep 10 23:55:39.587335 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.587178 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb6a6aeb41f622577c4973d1624f83e785590e62fb33849b5ddf05ef8162a281" Sep 10 23:55:39.624534 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.624511 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:55:39.727125 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727066 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istiod-ca-cert\") pod \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " Sep 10 23:55:39.727527 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727155 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-workload-socket\") pod \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " Sep 10 23:55:39.727527 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727182 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-envoy\") pod \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " Sep 10 23:55:39.727527 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727231 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-credential-socket\") pod \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " Sep 10 23:55:39.727527 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727315 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-podinfo\") pod \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " Sep 10 23:55:39.727527 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727343 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-data\") pod \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " Sep 10 23:55:39.727527 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727380 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq7mz\" (UniqueName: \"kubernetes.io/projected/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-kube-api-access-fq7mz\") pod \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " Sep 10 23:55:39.727527 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727464 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2a05d0e-cf4f-46ae-8230-222f9b197d5c" (UID: "e2a05d0e-cf4f-46ae-8230-222f9b197d5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:55:39.727527 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727471 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2a05d0e-cf4f-46ae-8230-222f9b197d5c" (UID: "e2a05d0e-cf4f-46ae-8230-222f9b197d5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:39.727527 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727528 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-workload-certs\") pod \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " Sep 10 23:55:39.728106 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727546 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2a05d0e-cf4f-46ae-8230-222f9b197d5c" (UID: "e2a05d0e-cf4f-46ae-8230-222f9b197d5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:39.728106 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727567 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-token\") pod \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\" (UID: \"e2a05d0e-cf4f-46ae-8230-222f9b197d5c\") " Sep 10 23:55:39.728106 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727693 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2a05d0e-cf4f-46ae-8230-222f9b197d5c" (UID: "e2a05d0e-cf4f-46ae-8230-222f9b197d5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:39.728106 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.727989 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2a05d0e-cf4f-46ae-8230-222f9b197d5c" (UID: "e2a05d0e-cf4f-46ae-8230-222f9b197d5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:39.729194 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.729175 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:39.729292 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.729201 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:39.729292 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.729216 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:39.729292 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.729230 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:39.729292 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.729243 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:39.729864 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.729831 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2a05d0e-cf4f-46ae-8230-222f9b197d5c" (UID: "e2a05d0e-cf4f-46ae-8230-222f9b197d5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:39.730228 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.730205 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-kube-api-access-fq7mz" (OuterVolumeSpecName: "kube-api-access-fq7mz") pod "e2a05d0e-cf4f-46ae-8230-222f9b197d5c" (UID: "e2a05d0e-cf4f-46ae-8230-222f9b197d5c"). InnerVolumeSpecName "kube-api-access-fq7mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:39.730327 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.730250 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2a05d0e-cf4f-46ae-8230-222f9b197d5c" (UID: "e2a05d0e-cf4f-46ae-8230-222f9b197d5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:39.730532 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.730514 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2a05d0e-cf4f-46ae-8230-222f9b197d5c" (UID: "e2a05d0e-cf4f-46ae-8230-222f9b197d5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:55:39.829837 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.829810 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:39.829837 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.829833 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:39.829837 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.829842 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:39.830026 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:39.829851 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq7mz\" (UniqueName: \"kubernetes.io/projected/e2a05d0e-cf4f-46ae-8230-222f9b197d5c-kube-api-access-fq7mz\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:40.593444 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:40.593412 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" Sep 10 23:55:40.593444 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:40.593429 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw" event={"ID":"e2a05d0e-cf4f-46ae-8230-222f9b197d5c","Type":"ContainerDied","Data":"90bbd2e4aea883fc1e69d1ff4ad97a46a2db39cfc34f3fe02acb36396f7060c7"} Sep 10 23:55:40.593660 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:40.593504 2583 scope.go:117] "RemoveContainer" containerID="19dcfe60166e3d1d088ecb3a46c6b638e66c80a68b0ff31f4ee65940fa8d8a5d" Sep 10 23:55:40.616709 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:40.616676 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw"] Sep 10 23:55:40.621238 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:40.621213 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-76b16f-openshift-default-5648b6569b-l6pvw"] Sep 10 23:55:41.307340 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:41.307299 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2a05d0e-cf4f-46ae-8230-222f9b197d5c" path="/var/lib/kubelet/pods/e2a05d0e-cf4f-46ae-8230-222f9b197d5c/volumes" Sep 10 23:55:45.027611 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.027579 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6"] Sep 10 23:55:45.027995 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.027926 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd1245a-79a4-4ead-8f39-99c2450eaa8b" containerName="revoke-keys" Sep 10 23:55:45.027995 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.027940 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd1245a-79a4-4ead-8f39-99c2450eaa8b" containerName="revoke-keys" Sep 10 23:55:45.027995 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.027952 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a05d0e-cf4f-46ae-8230-222f9b197d5c" containerName="istio-proxy" Sep 10 23:55:45.027995 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.027957 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a05d0e-cf4f-46ae-8230-222f9b197d5c" containerName="istio-proxy" Sep 10 23:55:45.028150 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.028009 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a05d0e-cf4f-46ae-8230-222f9b197d5c" containerName="istio-proxy" Sep 10 23:55:45.028150 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.028019 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd1245a-79a4-4ead-8f39-99c2450eaa8b" containerName="revoke-keys" Sep 10 23:55:45.032378 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.032360 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" Sep 10 23:55:45.034878 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.034854 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-16e45a\"" Sep 10 23:55:45.035343 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.035328 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-16e45a-dockercfg-hq95q\"" Sep 10 23:55:45.043498 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.043471 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6"] Sep 10 23:55:45.072461 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.072430 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b61aec6-a470-4f6a-a4b0-a7c388feebac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6\" (UID: \"7b61aec6-a470-4f6a-a4b0-a7c388feebac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" Sep 10 23:55:45.072616 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.072565 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-667nq\" (UniqueName: \"kubernetes.io/projected/7b61aec6-a470-4f6a-a4b0-a7c388feebac-kube-api-access-667nq\") pod \"payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6\" (UID: \"7b61aec6-a470-4f6a-a4b0-a7c388feebac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" Sep 10 23:55:45.072936 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.072918 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2"] Sep 10 23:55:45.076043 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.076028 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" Sep 10 23:55:45.086753 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.086731 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2"] Sep 10 23:55:45.173745 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.173707 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b61aec6-a470-4f6a-a4b0-a7c388feebac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6\" (UID: \"7b61aec6-a470-4f6a-a4b0-a7c388feebac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" Sep 10 23:55:45.173919 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.173770 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cc0c12c-3aa3-4b95-895c-088d21f03987-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2\" (UID: \"7cc0c12c-3aa3-4b95-895c-088d21f03987\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" Sep 10 23:55:45.173919 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.173817 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667nq\" (UniqueName: \"kubernetes.io/projected/7b61aec6-a470-4f6a-a4b0-a7c388feebac-kube-api-access-667nq\") pod \"payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6\" (UID: \"7b61aec6-a470-4f6a-a4b0-a7c388feebac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" Sep 10 23:55:45.173919 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.173841 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmpm\" (UniqueName: \"kubernetes.io/projected/7cc0c12c-3aa3-4b95-895c-088d21f03987-kube-api-access-lvmpm\") pod \"payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2\" (UID: \"7cc0c12c-3aa3-4b95-895c-088d21f03987\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" Sep 10 23:55:45.174488 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.174468 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b61aec6-a470-4f6a-a4b0-a7c388feebac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6\" (UID: \"7b61aec6-a470-4f6a-a4b0-a7c388feebac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" Sep 10 23:55:45.181617 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.181589 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-667nq\" (UniqueName: \"kubernetes.io/projected/7b61aec6-a470-4f6a-a4b0-a7c388feebac-kube-api-access-667nq\") pod \"payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6\" (UID: \"7b61aec6-a470-4f6a-a4b0-a7c388feebac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" Sep 10 23:55:45.274818 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.274789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmpm\" (UniqueName: \"kubernetes.io/projected/7cc0c12c-3aa3-4b95-895c-088d21f03987-kube-api-access-lvmpm\") pod \"payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2\" (UID: \"7cc0c12c-3aa3-4b95-895c-088d21f03987\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" Sep 10 23:55:45.274956 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.274874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cc0c12c-3aa3-4b95-895c-088d21f03987-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2\" (UID: \"7cc0c12c-3aa3-4b95-895c-088d21f03987\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" Sep 10 23:55:45.275511 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.275494 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cc0c12c-3aa3-4b95-895c-088d21f03987-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2\" (UID: \"7cc0c12c-3aa3-4b95-895c-088d21f03987\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" Sep 10 23:55:45.283095 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.283016 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmpm\" (UniqueName: \"kubernetes.io/projected/7cc0c12c-3aa3-4b95-895c-088d21f03987-kube-api-access-lvmpm\") pod \"payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2\" (UID: \"7cc0c12c-3aa3-4b95-895c-088d21f03987\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" Sep 10 23:55:45.346542 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.346520 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" Sep 10 23:55:45.386202 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.386171 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" Sep 10 23:55:45.625618 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.625433 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6"] Sep 10 23:55:45.628529 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:55:45.628501 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b61aec6_a470_4f6a_a4b0_a7c388feebac.slice/crio-f47ad3b88bdad557f54ee9a8bb834cc661cf572d685fcfd1797ec150eb5f16c6 WatchSource:0}: Error finding container f47ad3b88bdad557f54ee9a8bb834cc661cf572d685fcfd1797ec150eb5f16c6: Status 404 returned error can't find the container with id f47ad3b88bdad557f54ee9a8bb834cc661cf572d685fcfd1797ec150eb5f16c6 Sep 10 23:55:45.668953 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.668929 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2"] Sep 10 23:55:45.670771 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:55:45.670742 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc0c12c_3aa3_4b95_895c_088d21f03987.slice/crio-4ce1790c71ed16303db1bcd92981a5faaa5a93a7ecee45592ff22a85c002186b WatchSource:0}: Error finding container 4ce1790c71ed16303db1bcd92981a5faaa5a93a7ecee45592ff22a85c002186b: Status 404 returned error can't find the container with id 4ce1790c71ed16303db1bcd92981a5faaa5a93a7ecee45592ff22a85c002186b Sep 10 23:55:45.849955 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.849919 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c"] Sep 10 23:55:45.851656 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:45.851634 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4c5699-wkx4c"] Sep 10 23:55:46.626339 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:46.626302 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" event={"ID":"7cc0c12c-3aa3-4b95-895c-088d21f03987","Type":"ContainerStarted","Data":"9186cf9549183c1be196287d12da136b003bfc45f2f2bbdd6bba400ae23c92ac"} Sep 10 23:55:46.626765 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:46.626454 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" Sep 10 23:55:46.626765 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:46.626499 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" event={"ID":"7cc0c12c-3aa3-4b95-895c-088d21f03987","Type":"ContainerStarted","Data":"4ce1790c71ed16303db1bcd92981a5faaa5a93a7ecee45592ff22a85c002186b"} Sep 10 23:55:46.627906 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:46.627885 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" event={"ID":"7b61aec6-a470-4f6a-a4b0-a7c388feebac","Type":"ContainerStarted","Data":"ff43b7bb881097cb9d794a63b0067668ac46260dc7395cf1c0dbedc49abd63e2"} Sep 10 23:55:46.628011 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:46.627925 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" event={"ID":"7b61aec6-a470-4f6a-a4b0-a7c388feebac","Type":"ContainerStarted","Data":"f47ad3b88bdad557f54ee9a8bb834cc661cf572d685fcfd1797ec150eb5f16c6"} Sep 10 23:55:46.628011 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:46.627960 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" Sep 10 23:55:46.642912 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:46.642863 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" podStartSLOduration=1.642845872 podStartE2EDuration="1.642845872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:46.642176128 +0000 UTC m=+1153.944654152" watchObservedRunningTime="2026-09-10 23:55:46.642845872 +0000 UTC m=+1153.945323876" Sep 10 23:55:46.657965 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:46.657922 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" podStartSLOduration=1.657909603 podStartE2EDuration="1.657909603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:46.656834049 +0000 UTC m=+1153.959312062" watchObservedRunningTime="2026-09-10 23:55:46.657909603 +0000 UTC m=+1153.960387615" Sep 10 23:55:47.308228 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:47.308185 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd1245a-79a4-4ead-8f39-99c2450eaa8b" path="/var/lib/kubelet/pods/ebd1245a-79a4-4ead-8f39-99c2450eaa8b/volumes" Sep 10 23:55:52.301472 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:52.301428 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj"] Sep 10 23:55:52.301937 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:52.301728 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" podUID="90546c99-e542-4446-931d-be95d2dbd11c" containerName="istio-proxy" containerID="cri-o://d750ac31d277136f1e2d923153356df389400ac60fd5c8b9e3b0f85044da392a" gracePeriod=30 Sep 10 23:55:52.659549 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:52.659515 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7"] Sep 10 23:55:52.663853 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:52.663837 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7" Sep 10 23:55:52.666185 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:52.666147 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 10 23:55:52.666317 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:52.666214 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 23:55:52.670830 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:52.670802 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7"] Sep 10 23:55:52.743539 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:52.743502 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbf10a5-d771-4e9a-9f54-7dff4d4ec735-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7\" (UID: \"ecbf10a5-d771-4e9a-9f54-7dff4d4ec735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7" Sep 10 23:55:52.844963 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:52.844924 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbf10a5-d771-4e9a-9f54-7dff4d4ec735-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7\" (UID: \"ecbf10a5-d771-4e9a-9f54-7dff4d4ec735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7" Sep 10 23:55:52.845582 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:52.845561 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbf10a5-d771-4e9a-9f54-7dff4d4ec735-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7\" (UID: \"ecbf10a5-d771-4e9a-9f54-7dff4d4ec735\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7" Sep 10 23:55:52.976372 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:52.976285 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7" Sep 10 23:55:53.246342 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:53.246319 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7"] Sep 10 23:55:53.247792 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:55:53.247762 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbf10a5_d771_4e9a_9f54_7dff4d4ec735.slice/crio-53c6083742c2018034ddd32eff5ae5cee037e6049c5f5a1d7752234ee009f6ef WatchSource:0}: Error finding container 53c6083742c2018034ddd32eff5ae5cee037e6049c5f5a1d7752234ee009f6ef: Status 404 returned error can't find the container with id 53c6083742c2018034ddd32eff5ae5cee037e6049c5f5a1d7752234ee009f6ef Sep 10 23:55:53.665685 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:53.665646 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7" event={"ID":"ecbf10a5-d771-4e9a-9f54-7dff4d4ec735","Type":"ContainerStarted","Data":"1d349e59bb60caa29235427689c311f9b6fbb3c02d3bdc1948423110e1222d9a"} Sep 10 23:55:53.666047 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:53.665698 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7" event={"ID":"ecbf10a5-d771-4e9a-9f54-7dff4d4ec735","Type":"ContainerStarted","Data":"53c6083742c2018034ddd32eff5ae5cee037e6049c5f5a1d7752234ee009f6ef"} Sep 10 23:55:53.683523 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:53.683472 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7" podStartSLOduration=1.6834587619999999 podStartE2EDuration="1.683458762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:53.681219857 +0000 UTC m=+1160.983697870" watchObservedRunningTime="2026-09-10 23:55:53.683458762 +0000 UTC m=+1160.985936774" Sep 10 23:55:54.672406 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:54.672371 2583 generic.go:356] "Generic (PLEG): container finished" podID="ecbf10a5-d771-4e9a-9f54-7dff4d4ec735" containerID="1d349e59bb60caa29235427689c311f9b6fbb3c02d3bdc1948423110e1222d9a" exitCode=0 Sep 10 23:55:54.672787 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:54.672455 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7" event={"ID":"ecbf10a5-d771-4e9a-9f54-7dff4d4ec735","Type":"ContainerDied","Data":"1d349e59bb60caa29235427689c311f9b6fbb3c02d3bdc1948423110e1222d9a"} Sep 10 23:55:55.800861 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:55.800837 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7" Sep 10 23:55:55.872337 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:55.872305 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbf10a5-d771-4e9a-9f54-7dff4d4ec735-maas-api-service-ca\") pod \"ecbf10a5-d771-4e9a-9f54-7dff4d4ec735\" (UID: \"ecbf10a5-d771-4e9a-9f54-7dff4d4ec735\") " Sep 10 23:55:55.872628 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:55.872607 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecbf10a5-d771-4e9a-9f54-7dff4d4ec735-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ecbf10a5-d771-4e9a-9f54-7dff4d4ec735" (UID: "ecbf10a5-d771-4e9a-9f54-7dff4d4ec735"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:55:55.973900 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:55.973824 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbf10a5-d771-4e9a-9f54-7dff4d4ec735-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:56.684193 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:56.684162 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7" event={"ID":"ecbf10a5-d771-4e9a-9f54-7dff4d4ec735","Type":"ContainerDied","Data":"53c6083742c2018034ddd32eff5ae5cee037e6049c5f5a1d7752234ee009f6ef"} Sep 10 23:55:56.684193 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:56.684193 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7" Sep 10 23:55:56.684410 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:56.684209 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53c6083742c2018034ddd32eff5ae5cee037e6049c5f5a1d7752234ee009f6ef" Sep 10 23:55:57.555504 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.555480 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:55:57.635169 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.635144 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" Sep 10 23:55:57.635333 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.635212 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" Sep 10 23:55:57.689522 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.689492 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-istio-data\") pod \"90546c99-e542-4446-931d-be95d2dbd11c\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " Sep 10 23:55:57.689687 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.689533 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld6mh\" (UniqueName: \"kubernetes.io/projected/90546c99-e542-4446-931d-be95d2dbd11c-kube-api-access-ld6mh\") pod \"90546c99-e542-4446-931d-be95d2dbd11c\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " Sep 10 23:55:57.689687 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.689567 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-workload-socket\") pod \"90546c99-e542-4446-931d-be95d2dbd11c\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " Sep 10 23:55:57.689687 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.689603 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-workload-certs\") pod \"90546c99-e542-4446-931d-be95d2dbd11c\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " Sep 10 23:55:57.689687 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.689629 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-istio-envoy\") pod \"90546c99-e542-4446-931d-be95d2dbd11c\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " Sep 10 23:55:57.689891 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.689689 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90546c99-e542-4446-931d-be95d2dbd11c-istio-podinfo\") pod \"90546c99-e542-4446-931d-be95d2dbd11c\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " Sep 10 23:55:57.689891 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.689732 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90546c99-e542-4446-931d-be95d2dbd11c-istio-token\") pod \"90546c99-e542-4446-931d-be95d2dbd11c\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " Sep 10 23:55:57.689891 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.689762 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-credential-socket\") pod \"90546c99-e542-4446-931d-be95d2dbd11c\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " Sep 10 23:55:57.689891 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.689787 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-istio-data" (OuterVolumeSpecName: "istio-data") pod "90546c99-e542-4446-931d-be95d2dbd11c" (UID: "90546c99-e542-4446-931d-be95d2dbd11c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:57.689891 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.689866 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90546c99-e542-4446-931d-be95d2dbd11c-istiod-ca-cert\") pod \"90546c99-e542-4446-931d-be95d2dbd11c\" (UID: \"90546c99-e542-4446-931d-be95d2dbd11c\") " Sep 10 23:55:57.690473 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.690159 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "90546c99-e542-4446-931d-be95d2dbd11c" (UID: "90546c99-e542-4446-931d-be95d2dbd11c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:57.690473 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.690240 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:57.690473 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.690261 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:57.690473 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.690396 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90546c99-e542-4446-931d-be95d2dbd11c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "90546c99-e542-4446-931d-be95d2dbd11c" (UID: "90546c99-e542-4446-931d-be95d2dbd11c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:55:57.690769 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.690498 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "90546c99-e542-4446-931d-be95d2dbd11c" (UID: "90546c99-e542-4446-931d-be95d2dbd11c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:57.690769 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.690650 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "90546c99-e542-4446-931d-be95d2dbd11c" (UID: "90546c99-e542-4446-931d-be95d2dbd11c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:57.691142 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.691113 2583 generic.go:356] "Generic (PLEG): container finished" podID="90546c99-e542-4446-931d-be95d2dbd11c" containerID="d750ac31d277136f1e2d923153356df389400ac60fd5c8b9e3b0f85044da392a" exitCode=0 Sep 10 23:55:57.691269 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.691249 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" event={"ID":"90546c99-e542-4446-931d-be95d2dbd11c","Type":"ContainerDied","Data":"d750ac31d277136f1e2d923153356df389400ac60fd5c8b9e3b0f85044da392a"} Sep 10 23:55:57.691335 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.691308 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" event={"ID":"90546c99-e542-4446-931d-be95d2dbd11c","Type":"ContainerDied","Data":"66fe843fbe4b573210042049ab8fb5d8f2b4caa15573ca5756de8b1452eb2654"} Sep 10 23:55:57.691335 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.691326 2583 scope.go:117] "RemoveContainer" containerID="d750ac31d277136f1e2d923153356df389400ac60fd5c8b9e3b0f85044da392a" Sep 10 23:55:57.691440 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.691308 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj" Sep 10 23:55:57.692592 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.692531 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "90546c99-e542-4446-931d-be95d2dbd11c" (UID: "90546c99-e542-4446-931d-be95d2dbd11c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:57.692974 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.692949 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90546c99-e542-4446-931d-be95d2dbd11c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "90546c99-e542-4446-931d-be95d2dbd11c" (UID: "90546c99-e542-4446-931d-be95d2dbd11c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:55:57.692974 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.692956 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90546c99-e542-4446-931d-be95d2dbd11c-kube-api-access-ld6mh" (OuterVolumeSpecName: "kube-api-access-ld6mh") pod "90546c99-e542-4446-931d-be95d2dbd11c" (UID: "90546c99-e542-4446-931d-be95d2dbd11c"). InnerVolumeSpecName "kube-api-access-ld6mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:57.693150 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.692963 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90546c99-e542-4446-931d-be95d2dbd11c-istio-token" (OuterVolumeSpecName: "istio-token") pod "90546c99-e542-4446-931d-be95d2dbd11c" (UID: "90546c99-e542-4446-931d-be95d2dbd11c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:57.710620 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.710600 2583 scope.go:117] "RemoveContainer" containerID="4b0ba308079182f0e0eb956e04d5d8479ba135526b3b8dc0481804400886ccca" Sep 10 23:55:57.718512 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.718487 2583 scope.go:117] "RemoveContainer" containerID="d750ac31d277136f1e2d923153356df389400ac60fd5c8b9e3b0f85044da392a" Sep 10 23:55:57.718782 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:55:57.718757 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d750ac31d277136f1e2d923153356df389400ac60fd5c8b9e3b0f85044da392a\": container with ID starting with d750ac31d277136f1e2d923153356df389400ac60fd5c8b9e3b0f85044da392a not found: ID does not exist" containerID="d750ac31d277136f1e2d923153356df389400ac60fd5c8b9e3b0f85044da392a" Sep 10 23:55:57.718853 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.718788 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d750ac31d277136f1e2d923153356df389400ac60fd5c8b9e3b0f85044da392a"} err="failed to get container status \"d750ac31d277136f1e2d923153356df389400ac60fd5c8b9e3b0f85044da392a\": rpc error: code = NotFound desc = could not find container \"d750ac31d277136f1e2d923153356df389400ac60fd5c8b9e3b0f85044da392a\": container with ID starting with d750ac31d277136f1e2d923153356df389400ac60fd5c8b9e3b0f85044da392a not found: ID does not exist" Sep 10 23:55:57.718853 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.718807 2583 scope.go:117] "RemoveContainer" containerID="4b0ba308079182f0e0eb956e04d5d8479ba135526b3b8dc0481804400886ccca" Sep 10 23:55:57.719131 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:55:57.719109 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0ba308079182f0e0eb956e04d5d8479ba135526b3b8dc0481804400886ccca\": container with ID starting with 4b0ba308079182f0e0eb956e04d5d8479ba135526b3b8dc0481804400886ccca not found: ID does not exist" containerID="4b0ba308079182f0e0eb956e04d5d8479ba135526b3b8dc0481804400886ccca" Sep 10 23:55:57.719187 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.719137 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0ba308079182f0e0eb956e04d5d8479ba135526b3b8dc0481804400886ccca"} err="failed to get container status \"4b0ba308079182f0e0eb956e04d5d8479ba135526b3b8dc0481804400886ccca\": rpc error: code = NotFound desc = could not find container \"4b0ba308079182f0e0eb956e04d5d8479ba135526b3b8dc0481804400886ccca\": container with ID starting with 4b0ba308079182f0e0eb956e04d5d8479ba135526b3b8dc0481804400886ccca not found: ID does not exist" Sep 10 23:55:57.791504 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.791473 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90546c99-e542-4446-931d-be95d2dbd11c-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:57.791504 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.791501 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld6mh\" (UniqueName: \"kubernetes.io/projected/90546c99-e542-4446-931d-be95d2dbd11c-kube-api-access-ld6mh\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:57.791504 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.791511 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:57.791735 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.791521 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:57.791735 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.791531 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90546c99-e542-4446-931d-be95d2dbd11c-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:57.791735 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.791539 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90546c99-e542-4446-931d-be95d2dbd11c-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:57.791735 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:57.791547 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90546c99-e542-4446-931d-be95d2dbd11c-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:55:58.014741 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:58.014694 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj"] Sep 10 23:55:58.017284 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:58.017257 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4c5699-openshift-default-7d96775c4-6rhsj"] Sep 10 23:55:59.307507 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:55:59.307477 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90546c99-e542-4446-931d-be95d2dbd11c" path="/var/lib/kubelet/pods/90546c99-e542-4446-931d-be95d2dbd11c/volumes" Sep 10 23:56:03.425374 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:03.425338 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7"] Sep 10 23:56:03.427203 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:03.427181 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-948e64-wvmk7"] Sep 10 23:56:05.307543 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:05.307510 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecbf10a5-d771-4e9a-9f54-7dff4d4ec735" path="/var/lib/kubelet/pods/ecbf10a5-d771-4e9a-9f54-7dff4d4ec735/volumes" Sep 10 23:56:09.935469 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:09.935432 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2"] Sep 10 23:56:09.935951 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:09.935679 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" podUID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerName="istio-proxy" containerID="cri-o://050d91f16e1974e3f0401ecc02d596d18584c595b2a2837758c5b45abf72c583" gracePeriod=30 Sep 10 23:56:15.181675 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.181647 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:56:15.340958 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.340870 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-workload-certs\") pod \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " Sep 10 23:56:15.340958 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.340944 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-workload-socket\") pod \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " Sep 10 23:56:15.341259 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.340969 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-credential-socket\") pod \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " Sep 10 23:56:15.341259 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.340998 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-token\") pod \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " Sep 10 23:56:15.341259 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341026 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istiod-ca-cert\") pod \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " Sep 10 23:56:15.341259 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341061 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-data\") pod \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " Sep 10 23:56:15.341259 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341109 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wllw\" (UniqueName: \"kubernetes.io/projected/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-kube-api-access-6wllw\") pod \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " Sep 10 23:56:15.341259 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341185 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4aeb6d3c-8e7c-4928-adac-1dc6749fa329" (UID: "4aeb6d3c-8e7c-4928-adac-1dc6749fa329"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:56:15.341259 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341243 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-envoy\") pod \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " Sep 10 23:56:15.341625 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341285 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-podinfo\") pod \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\" (UID: \"4aeb6d3c-8e7c-4928-adac-1dc6749fa329\") " Sep 10 23:56:15.341625 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341343 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-data" (OuterVolumeSpecName: "istio-data") pod "4aeb6d3c-8e7c-4928-adac-1dc6749fa329" (UID: "4aeb6d3c-8e7c-4928-adac-1dc6749fa329"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:56:15.341625 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341360 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4aeb6d3c-8e7c-4928-adac-1dc6749fa329" (UID: "4aeb6d3c-8e7c-4928-adac-1dc6749fa329"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:56:15.341625 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341379 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4aeb6d3c-8e7c-4928-adac-1dc6749fa329" (UID: "4aeb6d3c-8e7c-4928-adac-1dc6749fa329"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:56:15.341625 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341507 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4aeb6d3c-8e7c-4928-adac-1dc6749fa329" (UID: "4aeb6d3c-8e7c-4928-adac-1dc6749fa329"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:56:15.341625 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341597 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:56:15.341625 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341617 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:56:15.341908 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341631 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:56:15.341908 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341644 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:56:15.341908 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.341656 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:56:15.343416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.343382 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-kube-api-access-6wllw" (OuterVolumeSpecName: "kube-api-access-6wllw") pod "4aeb6d3c-8e7c-4928-adac-1dc6749fa329" (UID: "4aeb6d3c-8e7c-4928-adac-1dc6749fa329"). InnerVolumeSpecName "kube-api-access-6wllw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:56:15.343416 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.343410 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4aeb6d3c-8e7c-4928-adac-1dc6749fa329" (UID: "4aeb6d3c-8e7c-4928-adac-1dc6749fa329"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:56:15.343649 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.343632 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4aeb6d3c-8e7c-4928-adac-1dc6749fa329" (UID: "4aeb6d3c-8e7c-4928-adac-1dc6749fa329"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:56:15.343747 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.343649 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-token" (OuterVolumeSpecName: "istio-token") pod "4aeb6d3c-8e7c-4928-adac-1dc6749fa329" (UID: "4aeb6d3c-8e7c-4928-adac-1dc6749fa329"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:56:15.442313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.442275 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wllw\" (UniqueName: \"kubernetes.io/projected/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-kube-api-access-6wllw\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:56:15.442313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.442306 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:56:15.442313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.442318 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:56:15.442313 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.442326 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aeb6d3c-8e7c-4928-adac-1dc6749fa329-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:56:15.785737 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.785698 2583 generic.go:356] "Generic (PLEG): container finished" podID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerID="050d91f16e1974e3f0401ecc02d596d18584c595b2a2837758c5b45abf72c583" exitCode=0 Sep 10 23:56:15.785922 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.785755 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" Sep 10 23:56:15.785922 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.785777 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" event={"ID":"4aeb6d3c-8e7c-4928-adac-1dc6749fa329","Type":"ContainerDied","Data":"050d91f16e1974e3f0401ecc02d596d18584c595b2a2837758c5b45abf72c583"} Sep 10 23:56:15.785922 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.785830 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2" event={"ID":"4aeb6d3c-8e7c-4928-adac-1dc6749fa329","Type":"ContainerDied","Data":"5247b1fc070a3660c5583b4b821da6d07afeba43729ba02f9682b1c1798ed246"} Sep 10 23:56:15.785922 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.785853 2583 scope.go:117] "RemoveContainer" containerID="050d91f16e1974e3f0401ecc02d596d18584c595b2a2837758c5b45abf72c583" Sep 10 23:56:15.795001 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.794984 2583 scope.go:117] "RemoveContainer" containerID="e44bdd3a75cd62689fb3ef973a10a817dbcbe55bb7f200691cc3e7451167ca4a" Sep 10 23:56:15.803241 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.803223 2583 scope.go:117] "RemoveContainer" containerID="050d91f16e1974e3f0401ecc02d596d18584c595b2a2837758c5b45abf72c583" Sep 10 23:56:15.803504 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:56:15.803482 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050d91f16e1974e3f0401ecc02d596d18584c595b2a2837758c5b45abf72c583\": container with ID starting with 050d91f16e1974e3f0401ecc02d596d18584c595b2a2837758c5b45abf72c583 not found: ID does not exist" containerID="050d91f16e1974e3f0401ecc02d596d18584c595b2a2837758c5b45abf72c583" Sep 10 23:56:15.803581 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.803517 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050d91f16e1974e3f0401ecc02d596d18584c595b2a2837758c5b45abf72c583"} err="failed to get container status \"050d91f16e1974e3f0401ecc02d596d18584c595b2a2837758c5b45abf72c583\": rpc error: code = NotFound desc = could not find container \"050d91f16e1974e3f0401ecc02d596d18584c595b2a2837758c5b45abf72c583\": container with ID starting with 050d91f16e1974e3f0401ecc02d596d18584c595b2a2837758c5b45abf72c583 not found: ID does not exist" Sep 10 23:56:15.803581 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.803543 2583 scope.go:117] "RemoveContainer" containerID="e44bdd3a75cd62689fb3ef973a10a817dbcbe55bb7f200691cc3e7451167ca4a" Sep 10 23:56:15.803803 ip-10-0-134-90 kubenswrapper[2583]: E0910 23:56:15.803787 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e44bdd3a75cd62689fb3ef973a10a817dbcbe55bb7f200691cc3e7451167ca4a\": container with ID starting with e44bdd3a75cd62689fb3ef973a10a817dbcbe55bb7f200691cc3e7451167ca4a not found: ID does not exist" containerID="e44bdd3a75cd62689fb3ef973a10a817dbcbe55bb7f200691cc3e7451167ca4a" Sep 10 23:56:15.803849 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.803808 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e44bdd3a75cd62689fb3ef973a10a817dbcbe55bb7f200691cc3e7451167ca4a"} err="failed to get container status \"e44bdd3a75cd62689fb3ef973a10a817dbcbe55bb7f200691cc3e7451167ca4a\": rpc error: code = NotFound desc = could not find container \"e44bdd3a75cd62689fb3ef973a10a817dbcbe55bb7f200691cc3e7451167ca4a\": container with ID starting with e44bdd3a75cd62689fb3ef973a10a817dbcbe55bb7f200691cc3e7451167ca4a not found: ID does not exist" Sep 10 23:56:15.808029 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.807997 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2"] Sep 10 23:56:15.812878 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:15.812858 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-948e64-openshift-default-7b8c6ddc84-bjzp2"] Sep 10 23:56:17.309893 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:17.309860 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" path="/var/lib/kubelet/pods/4aeb6d3c-8e7c-4928-adac-1dc6749fa329/volumes" Sep 10 23:56:19.972404 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.972368 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5"] Sep 10 23:56:19.972778 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.972682 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90546c99-e542-4446-931d-be95d2dbd11c" containerName="istio-proxy" Sep 10 23:56:19.972778 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.972693 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="90546c99-e542-4446-931d-be95d2dbd11c" containerName="istio-proxy" Sep 10 23:56:19.972778 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.972706 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecbf10a5-d771-4e9a-9f54-7dff4d4ec735" containerName="revoke-keys" Sep 10 23:56:19.972778 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.972711 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecbf10a5-d771-4e9a-9f54-7dff4d4ec735" containerName="revoke-keys" Sep 10 23:56:19.972778 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.972718 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerName="istio-proxy" Sep 10 23:56:19.972778 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.972724 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerName="istio-proxy" Sep 10 23:56:19.972778 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.972779 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="90546c99-e542-4446-931d-be95d2dbd11c" containerName="istio-proxy" Sep 10 23:56:19.973001 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.972787 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="90546c99-e542-4446-931d-be95d2dbd11c" containerName="istio-proxy" Sep 10 23:56:19.973001 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.972794 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerName="istio-proxy" Sep 10 23:56:19.973001 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.972800 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerName="istio-proxy" Sep 10 23:56:19.973001 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.972806 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecbf10a5-d771-4e9a-9f54-7dff4d4ec735" containerName="revoke-keys" Sep 10 23:56:19.977061 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.977040 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" Sep 10 23:56:19.979574 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.979552 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f8d9df10\"" Sep 10 23:56:19.979691 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.979598 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f8d9df10-dockercfg-vg795\"" Sep 10 23:56:19.987344 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:19.987322 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5"] Sep 10 23:56:20.087427 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.087391 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94274df-30a2-4996-920f-f991e35aa6d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5\" (UID: \"a94274df-30a2-4996-920f-f991e35aa6d2\") " pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" Sep 10 23:56:20.087599 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.087449 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njb8p\" (UniqueName: \"kubernetes.io/projected/a94274df-30a2-4996-920f-f991e35aa6d2-kube-api-access-njb8p\") pod \"payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5\" (UID: \"a94274df-30a2-4996-920f-f991e35aa6d2\") " pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" Sep 10 23:56:20.188850 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.188811 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njb8p\" (UniqueName: \"kubernetes.io/projected/a94274df-30a2-4996-920f-f991e35aa6d2-kube-api-access-njb8p\") pod \"payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5\" (UID: \"a94274df-30a2-4996-920f-f991e35aa6d2\") " pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" Sep 10 23:56:20.189027 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.188915 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94274df-30a2-4996-920f-f991e35aa6d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5\" (UID: \"a94274df-30a2-4996-920f-f991e35aa6d2\") " pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" Sep 10 23:56:20.189802 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.189776 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94274df-30a2-4996-920f-f991e35aa6d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5\" (UID: \"a94274df-30a2-4996-920f-f991e35aa6d2\") " pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" Sep 10 23:56:20.199022 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.198995 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njb8p\" (UniqueName: \"kubernetes.io/projected/a94274df-30a2-4996-920f-f991e35aa6d2-kube-api-access-njb8p\") pod \"payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5\" (UID: \"a94274df-30a2-4996-920f-f991e35aa6d2\") " pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" Sep 10 23:56:20.287722 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.287631 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" Sep 10 23:56:20.562212 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.562184 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5"] Sep 10 23:56:20.564050 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:56:20.564019 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94274df_30a2_4996_920f_f991e35aa6d2.slice/crio-128849c6cc28d7fd188cf9e28da3a9a38004c145eefba4e652e65d90a0ea1ec1 WatchSource:0}: Error finding container 128849c6cc28d7fd188cf9e28da3a9a38004c145eefba4e652e65d90a0ea1ec1: Status 404 returned error can't find the container with id 128849c6cc28d7fd188cf9e28da3a9a38004c145eefba4e652e65d90a0ea1ec1 Sep 10 23:56:20.813476 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.813385 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" event={"ID":"a94274df-30a2-4996-920f-f991e35aa6d2","Type":"ContainerStarted","Data":"bae8f3bc1b0fcb63c2977f60df334f89ea4532470c614daaa9fc6799be620bcf"} Sep 10 23:56:20.813476 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.813451 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" event={"ID":"a94274df-30a2-4996-920f-f991e35aa6d2","Type":"ContainerStarted","Data":"128849c6cc28d7fd188cf9e28da3a9a38004c145eefba4e652e65d90a0ea1ec1"} Sep 10 23:56:20.813669 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.813479 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" Sep 10 23:56:20.834823 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.834768 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" podStartSLOduration=1.83474779 podStartE2EDuration="1.83474779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:20.833819713 +0000 UTC m=+1188.136297727" watchObservedRunningTime="2026-09-10 23:56:20.83474779 +0000 UTC m=+1188.137225804" Sep 10 23:56:20.927398 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.927360 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr"] Sep 10 23:56:20.927838 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.927823 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90546c99-e542-4446-931d-be95d2dbd11c" containerName="istio-proxy" Sep 10 23:56:20.927890 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.927842 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="90546c99-e542-4446-931d-be95d2dbd11c" containerName="istio-proxy" Sep 10 23:56:20.927890 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.927862 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerName="istio-proxy" Sep 10 23:56:20.927890 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.927874 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aeb6d3c-8e7c-4928-adac-1dc6749fa329" containerName="istio-proxy" Sep 10 23:56:20.931183 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.931166 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" Sep 10 23:56:20.933607 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.933583 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-6fa41e\"" Sep 10 23:56:20.933935 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.933583 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-6fa41e-dockercfg-jph28\"" Sep 10 23:56:20.942691 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:20.942665 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr"] Sep 10 23:56:21.098103 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:21.097995 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a63268-3b36-46a6-86d4-62efd768c9da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr\" (UID: \"e9a63268-3b36-46a6-86d4-62efd768c9da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" Sep 10 23:56:21.098103 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:21.098033 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5tg5\" (UniqueName: \"kubernetes.io/projected/e9a63268-3b36-46a6-86d4-62efd768c9da-kube-api-access-r5tg5\") pod \"payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr\" (UID: \"e9a63268-3b36-46a6-86d4-62efd768c9da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" Sep 10 23:56:21.199690 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:21.199649 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a63268-3b36-46a6-86d4-62efd768c9da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr\" (UID: \"e9a63268-3b36-46a6-86d4-62efd768c9da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" Sep 10 23:56:21.199906 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:21.199697 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5tg5\" (UniqueName: \"kubernetes.io/projected/e9a63268-3b36-46a6-86d4-62efd768c9da-kube-api-access-r5tg5\") pod \"payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr\" (UID: \"e9a63268-3b36-46a6-86d4-62efd768c9da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" Sep 10 23:56:21.200451 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:21.200427 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a63268-3b36-46a6-86d4-62efd768c9da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr\" (UID: \"e9a63268-3b36-46a6-86d4-62efd768c9da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" Sep 10 23:56:21.209728 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:21.209701 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5tg5\" (UniqueName: \"kubernetes.io/projected/e9a63268-3b36-46a6-86d4-62efd768c9da-kube-api-access-r5tg5\") pod \"payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr\" (UID: \"e9a63268-3b36-46a6-86d4-62efd768c9da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" Sep 10 23:56:21.241226 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:21.241190 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" Sep 10 23:56:21.510871 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:21.510812 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr"] Sep 10 23:56:21.513212 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:56:21.513178 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a63268_3b36_46a6_86d4_62efd768c9da.slice/crio-86a92ce3de910d6811579728d6a65e6e8752a7641c21d98b64c3215e3dbb86d1 WatchSource:0}: Error finding container 86a92ce3de910d6811579728d6a65e6e8752a7641c21d98b64c3215e3dbb86d1: Status 404 returned error can't find the container with id 86a92ce3de910d6811579728d6a65e6e8752a7641c21d98b64c3215e3dbb86d1 Sep 10 23:56:21.820148 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:21.820031 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" event={"ID":"e9a63268-3b36-46a6-86d4-62efd768c9da","Type":"ContainerStarted","Data":"e7d1d153d9e6801281e42495a8b5431ac1aa6353b049674f883c0f6f12a8e498"} Sep 10 23:56:21.820148 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:21.820115 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" event={"ID":"e9a63268-3b36-46a6-86d4-62efd768c9da","Type":"ContainerStarted","Data":"86a92ce3de910d6811579728d6a65e6e8752a7641c21d98b64c3215e3dbb86d1"} Sep 10 23:56:21.836905 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:21.836840 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" podStartSLOduration=1.8368188970000001 podStartE2EDuration="1.836818897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:21.83582324 +0000 UTC m=+1189.138301252" watchObservedRunningTime="2026-09-10 23:56:21.836818897 +0000 UTC m=+1189.139296912" Sep 10 23:56:22.825112 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:22.825065 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" Sep 10 23:56:31.822701 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:31.822670 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" Sep 10 23:56:33.270359 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:33.270325 2583 scope.go:117] "RemoveContainer" containerID="8d44d5683bf4c8e8e366751ad670447cfcbc6a971a55cb355e673e179268756b" Sep 10 23:56:33.832761 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:33.832732 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" Sep 10 23:56:47.924714 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:47.924679 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz"] Sep 10 23:56:47.929514 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:47.929495 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:47.931935 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:47.931915 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-37c598-openshift-default-dockercfg-bsbxr\"" Sep 10 23:56:47.940940 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:47.940919 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz"] Sep 10 23:56:48.036421 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.036378 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28b6070-193d-4086-ba4c-b679fd2c84d4-istiod-ca-cert\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.036609 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.036454 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5chvd\" (UniqueName: \"kubernetes.io/projected/f28b6070-193d-4086-ba4c-b679fd2c84d4-kube-api-access-5chvd\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.036609 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.036517 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-workload-socket\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.036609 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.036602 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-workload-certs\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.036777 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.036631 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-credential-socket\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.036777 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.036665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-data\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.036777 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.036695 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-token\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.036777 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.036743 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-envoy\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.036777 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.036769 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-podinfo\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.137898 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.137860 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5chvd\" (UniqueName: \"kubernetes.io/projected/f28b6070-193d-4086-ba4c-b679fd2c84d4-kube-api-access-5chvd\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.137898 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.137904 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-workload-socket\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.138165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.137957 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-workload-certs\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.138165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.137976 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-credential-socket\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.138165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.138003 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-data\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.138165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.138028 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-token\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.138165 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.138116 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-envoy\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.138417 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.138164 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-podinfo\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.138417 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.138203 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28b6070-193d-4086-ba4c-b679fd2c84d4-istiod-ca-cert\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.138529 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.138469 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-workload-certs\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.138529 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.138512 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-data\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.138639 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.138547 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-credential-socket\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.138639 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.138584 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-workload-socket\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.138923 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.138902 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28b6070-193d-4086-ba4c-b679fd2c84d4-istiod-ca-cert\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.140549 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.140528 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-envoy\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.140760 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.140742 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-podinfo\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.145560 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.145535 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-token\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.145700 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.145683 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5chvd\" (UniqueName: \"kubernetes.io/projected/f28b6070-193d-4086-ba4c-b679fd2c84d4-kube-api-access-5chvd\") pod \"e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.241137 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.241061 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:48.523494 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.523468 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz"] Sep 10 23:56:48.525538 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:56:48.525510 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28b6070_193d_4086_ba4c_b679fd2c84d4.slice/crio-d126f321d065f0100379f3452b41823936630839cfe018c64755113e1a690a67 WatchSource:0}: Error finding container d126f321d065f0100379f3452b41823936630839cfe018c64755113e1a690a67: Status 404 returned error can't find the container with id d126f321d065f0100379f3452b41823936630839cfe018c64755113e1a690a67 Sep 10 23:56:48.527838 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.527804 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:56:48.527937 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.527886 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:56:48.528086 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.527953 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:56:48.962680 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.962641 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" event={"ID":"f28b6070-193d-4086-ba4c-b679fd2c84d4","Type":"ContainerStarted","Data":"11a5428f83e80302a7644b0f448a32e5bd0ce1c6f3696e9b72c31eaca267f104"} Sep 10 23:56:48.963217 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.962699 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" event={"ID":"f28b6070-193d-4086-ba4c-b679fd2c84d4","Type":"ContainerStarted","Data":"d126f321d065f0100379f3452b41823936630839cfe018c64755113e1a690a67"} Sep 10 23:56:48.986372 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:48.986324 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" podStartSLOduration=1.9863093109999999 podStartE2EDuration="1.986309311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:48.984289879 +0000 UTC m=+1216.286767892" watchObservedRunningTime="2026-09-10 23:56:48.986309311 +0000 UTC m=+1216.288787323" Sep 10 23:56:49.241649 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:49.241565 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:49.246797 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:49.246774 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:49.967782 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:49.967749 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:49.968674 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:49.968655 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:56:54.321676 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.321640 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd"] Sep 10 23:56:54.325388 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.325371 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" Sep 10 23:56:54.327905 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.327884 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-37c598-dockercfg-6tnhr\"" Sep 10 23:56:54.328164 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.328145 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-37c598\"" Sep 10 23:56:54.336763 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.336744 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd"] Sep 10 23:56:54.356656 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.356631 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6"] Sep 10 23:56:54.360002 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.359984 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" Sep 10 23:56:54.370665 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.370642 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6"] Sep 10 23:56:54.396005 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.395968 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2mlg\" (UniqueName: \"kubernetes.io/projected/bd428f77-aecd-4b65-9770-577af9e103a3-kube-api-access-l2mlg\") pod \"payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd\" (UID: \"bd428f77-aecd-4b65-9770-577af9e103a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" Sep 10 23:56:54.396135 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.396046 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd428f77-aecd-4b65-9770-577af9e103a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd\" (UID: \"bd428f77-aecd-4b65-9770-577af9e103a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" Sep 10 23:56:54.497174 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.497127 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd428f77-aecd-4b65-9770-577af9e103a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd\" (UID: \"bd428f77-aecd-4b65-9770-577af9e103a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" Sep 10 23:56:54.497361 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.497214 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsxj8\" (UniqueName: \"kubernetes.io/projected/ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846-kube-api-access-lsxj8\") pod \"payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6\" (UID: \"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" Sep 10 23:56:54.497361 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.497248 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6\" (UID: \"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" Sep 10 23:56:54.497455 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.497355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2mlg\" (UniqueName: \"kubernetes.io/projected/bd428f77-aecd-4b65-9770-577af9e103a3-kube-api-access-l2mlg\") pod \"payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd\" (UID: \"bd428f77-aecd-4b65-9770-577af9e103a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" Sep 10 23:56:54.497816 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.497796 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd428f77-aecd-4b65-9770-577af9e103a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd\" (UID: \"bd428f77-aecd-4b65-9770-577af9e103a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" Sep 10 23:56:54.505241 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.505214 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2mlg\" (UniqueName: \"kubernetes.io/projected/bd428f77-aecd-4b65-9770-577af9e103a3-kube-api-access-l2mlg\") pod \"payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd\" (UID: \"bd428f77-aecd-4b65-9770-577af9e103a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" Sep 10 23:56:54.598005 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.597940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsxj8\" (UniqueName: \"kubernetes.io/projected/ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846-kube-api-access-lsxj8\") pod \"payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6\" (UID: \"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" Sep 10 23:56:54.598005 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.597969 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6\" (UID: \"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" Sep 10 23:56:54.598629 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.598612 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6\" (UID: \"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" Sep 10 23:56:54.605662 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.605639 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsxj8\" (UniqueName: \"kubernetes.io/projected/ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846-kube-api-access-lsxj8\") pod \"payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6\" (UID: \"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" Sep 10 23:56:54.636517 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.636494 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" Sep 10 23:56:54.669794 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.669766 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" Sep 10 23:56:54.908833 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.908805 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd"] Sep 10 23:56:54.910350 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:56:54.910325 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd428f77_aecd_4b65_9770_577af9e103a3.slice/crio-ad73b943763ea306bdcc9409edc0b78c91ae8a3b634016f1bececd2b52825ccc WatchSource:0}: Error finding container ad73b943763ea306bdcc9409edc0b78c91ae8a3b634016f1bececd2b52825ccc: Status 404 returned error can't find the container with id ad73b943763ea306bdcc9409edc0b78c91ae8a3b634016f1bececd2b52825ccc Sep 10 23:56:54.949029 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.949003 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6"] Sep 10 23:56:54.951020 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:56:54.950987 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3b3fd3_2a3d_4d55_97bd_0a1b62daf846.slice/crio-0bcc07c0598d4b0a4b0ba2543d7343bcc71c49a22208cd18596c3ca9c2d639a7 WatchSource:0}: Error finding container 0bcc07c0598d4b0a4b0ba2543d7343bcc71c49a22208cd18596c3ca9c2d639a7: Status 404 returned error can't find the container with id 0bcc07c0598d4b0a4b0ba2543d7343bcc71c49a22208cd18596c3ca9c2d639a7 Sep 10 23:56:54.995500 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.995467 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" event={"ID":"bd428f77-aecd-4b65-9770-577af9e103a3","Type":"ContainerStarted","Data":"a975bd82f904101c4e6e522fc45de45fe74de7144704b2bc0858a437eba267a4"} Sep 10 23:56:54.995646 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.995535 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" Sep 10 23:56:54.995646 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.995564 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" event={"ID":"bd428f77-aecd-4b65-9770-577af9e103a3","Type":"ContainerStarted","Data":"ad73b943763ea306bdcc9409edc0b78c91ae8a3b634016f1bececd2b52825ccc"} Sep 10 23:56:54.996676 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:54.996656 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" event={"ID":"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846","Type":"ContainerStarted","Data":"0bcc07c0598d4b0a4b0ba2543d7343bcc71c49a22208cd18596c3ca9c2d639a7"} Sep 10 23:56:55.012749 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:55.012709 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" podStartSLOduration=1.012695628 podStartE2EDuration="1.012695628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:55.010642139 +0000 UTC m=+1222.313120152" watchObservedRunningTime="2026-09-10 23:56:55.012695628 +0000 UTC m=+1222.315173640" Sep 10 23:56:56.003809 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:56.003767 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" event={"ID":"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846","Type":"ContainerStarted","Data":"f05478f5cd6745813fb5f7fb08556709cd03e804d8eb7e4977edbb92fbc0fe42"} Sep 10 23:56:56.004271 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:56.003892 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" Sep 10 23:56:56.021402 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:56:56.021357 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" podStartSLOduration=2.021341237 podStartE2EDuration="2.021341237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:56.019654845 +0000 UTC m=+1223.322132876" watchObservedRunningTime="2026-09-10 23:56:56.021341237 +0000 UTC m=+1223.323819273" Sep 10 23:57:06.006151 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:06.006119 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" Sep 10 23:57:07.014384 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:07.014349 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" Sep 10 23:57:10.087385 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:10.087351 2583 generic.go:356] "Generic (PLEG): container finished" podID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerID="11a5428f83e80302a7644b0f448a32e5bd0ce1c6f3696e9b72c31eaca267f104" exitCode=0 Sep 10 23:57:10.087385 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:10.087387 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" event={"ID":"f28b6070-193d-4086-ba4c-b679fd2c84d4","Type":"ContainerDied","Data":"11a5428f83e80302a7644b0f448a32e5bd0ce1c6f3696e9b72c31eaca267f104"} Sep 10 23:57:10.087829 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:10.087757 2583 scope.go:117] "RemoveContainer" containerID="11a5428f83e80302a7644b0f448a32e5bd0ce1c6f3696e9b72c31eaca267f104" Sep 10 23:57:10.088902 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:10.088873 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:57:10.088965 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:10.088941 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:57:10.089002 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:10.088976 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:57:11.094040 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:11.094001 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" event={"ID":"f28b6070-193d-4086-ba4c-b679fd2c84d4","Type":"ContainerStarted","Data":"738f52849bdc5dc1aa6cb673e634fb605bf98881b8cb5d1393ff6bae4b2688b5"} Sep 10 23:57:11.241732 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:11.241332 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:57:11.242057 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:11.242031 2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 23:57:11.242280 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:11.242253 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 23:57:12.242494 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:12.242447 2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 23:57:12.242982 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:12.242532 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 23:57:13.242232 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:13.241737 2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 23:57:13.242232 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:13.241809 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 23:57:14.242606 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:14.242562 2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 23:57:14.243210 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:14.243166 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 23:57:15.241849 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:15.241810 2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 23:57:15.242171 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:15.242139 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 23:57:16.242092 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:16.242043 2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 23:57:16.242602 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:16.242143 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 23:57:16.281951 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:16.281918 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x"] Sep 10 23:57:16.285405 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:16.285385 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x" Sep 10 23:57:16.287542 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:16.287513 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 23:57:16.287716 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:16.287697 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 10 23:57:16.292866 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:16.292653 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x"] Sep 10 23:57:16.390202 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:16.390169 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/205a63cb-9162-44f5-9f55-91618e82bf17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x\" (UID: \"205a63cb-9162-44f5-9f55-91618e82bf17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x" Sep 10 23:57:16.490840 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:16.490805 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/205a63cb-9162-44f5-9f55-91618e82bf17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x\" (UID: \"205a63cb-9162-44f5-9f55-91618e82bf17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x" Sep 10 23:57:16.491520 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:16.491492 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/205a63cb-9162-44f5-9f55-91618e82bf17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x\" (UID: \"205a63cb-9162-44f5-9f55-91618e82bf17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x" Sep 10 23:57:16.596579 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:16.596498 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x" Sep 10 23:57:16.868716 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:16.868663 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x"] Sep 10 23:57:16.870953 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:57:16.870920 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205a63cb_9162_44f5_9f55_91618e82bf17.slice/crio-710fcf6163f27e5c2d5007866025bab49702ac9dc14783621ec5f640228ce04a WatchSource:0}: Error finding container 710fcf6163f27e5c2d5007866025bab49702ac9dc14783621ec5f640228ce04a: Status 404 returned error can't find the container with id 710fcf6163f27e5c2d5007866025bab49702ac9dc14783621ec5f640228ce04a Sep 10 23:57:17.125827 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:17.125795 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x" event={"ID":"205a63cb-9162-44f5-9f55-91618e82bf17","Type":"ContainerStarted","Data":"710fcf6163f27e5c2d5007866025bab49702ac9dc14783621ec5f640228ce04a"} Sep 10 23:57:17.245410 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:17.245378 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:57:17.245786 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:17.245631 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:57:17.246465 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:17.246445 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 10 23:57:18.133585 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:18.133544 2583 generic.go:356] "Generic (PLEG): container finished" podID="205a63cb-9162-44f5-9f55-91618e82bf17" containerID="62c0700334384f5c636f8d2d5012357bfafa85bc49586f7f0b7737c9774ae287" exitCode=0 Sep 10 23:57:18.133816 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:18.133630 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x" event={"ID":"205a63cb-9162-44f5-9f55-91618e82bf17","Type":"ContainerDied","Data":"62c0700334384f5c636f8d2d5012357bfafa85bc49586f7f0b7737c9774ae287"} Sep 10 23:57:19.268127 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:19.268101 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x" Sep 10 23:57:19.421097 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:19.420993 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/205a63cb-9162-44f5-9f55-91618e82bf17-maas-api-service-ca\") pod \"205a63cb-9162-44f5-9f55-91618e82bf17\" (UID: \"205a63cb-9162-44f5-9f55-91618e82bf17\") " Sep 10 23:57:19.421342 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:19.421319 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/205a63cb-9162-44f5-9f55-91618e82bf17-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "205a63cb-9162-44f5-9f55-91618e82bf17" (UID: "205a63cb-9162-44f5-9f55-91618e82bf17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:57:19.522372 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:19.522335 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/205a63cb-9162-44f5-9f55-91618e82bf17-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:57:20.145785 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:20.145747 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x" Sep 10 23:57:20.145785 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:20.145779 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x" event={"ID":"205a63cb-9162-44f5-9f55-91618e82bf17","Type":"ContainerDied","Data":"710fcf6163f27e5c2d5007866025bab49702ac9dc14783621ec5f640228ce04a"} Sep 10 23:57:20.146010 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:20.145832 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="710fcf6163f27e5c2d5007866025bab49702ac9dc14783621ec5f640228ce04a" Sep 10 23:57:32.036875 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:32.036787 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5"] Sep 10 23:57:32.037452 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:32.037019 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" podUID="a94274df-30a2-4996-920f-f991e35aa6d2" containerName="payload-processing" containerID="cri-o://bae8f3bc1b0fcb63c2977f60df334f89ea4532470c614daaa9fc6799be620bcf" gracePeriod=30 Sep 10 23:57:32.197656 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:32.197618 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x"] Sep 10 23:57:32.202925 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:32.202893 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f8d9df10-7wj2x"] Sep 10 23:57:32.211448 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:32.211406 2583 generic.go:356] "Generic (PLEG): container finished" podID="a94274df-30a2-4996-920f-f991e35aa6d2" containerID="bae8f3bc1b0fcb63c2977f60df334f89ea4532470c614daaa9fc6799be620bcf" exitCode=0 Sep 10 23:57:32.211594 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:32.211480 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" event={"ID":"a94274df-30a2-4996-920f-f991e35aa6d2","Type":"ContainerDied","Data":"bae8f3bc1b0fcb63c2977f60df334f89ea4532470c614daaa9fc6799be620bcf"} Sep 10 23:57:32.292111 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:32.291999 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" Sep 10 23:57:32.330842 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:32.330811 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94274df-30a2-4996-920f-f991e35aa6d2-plugins-config-volume\") pod \"a94274df-30a2-4996-920f-f991e35aa6d2\" (UID: \"a94274df-30a2-4996-920f-f991e35aa6d2\") " Sep 10 23:57:32.331013 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:32.330878 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njb8p\" (UniqueName: \"kubernetes.io/projected/a94274df-30a2-4996-920f-f991e35aa6d2-kube-api-access-njb8p\") pod \"a94274df-30a2-4996-920f-f991e35aa6d2\" (UID: \"a94274df-30a2-4996-920f-f991e35aa6d2\") " Sep 10 23:57:32.331329 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:32.331295 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94274df-30a2-4996-920f-f991e35aa6d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a94274df-30a2-4996-920f-f991e35aa6d2" (UID: "a94274df-30a2-4996-920f-f991e35aa6d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:57:32.332926 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:32.332900 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94274df-30a2-4996-920f-f991e35aa6d2-kube-api-access-njb8p" (OuterVolumeSpecName: "kube-api-access-njb8p") pod "a94274df-30a2-4996-920f-f991e35aa6d2" (UID: "a94274df-30a2-4996-920f-f991e35aa6d2"). InnerVolumeSpecName "kube-api-access-njb8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:57:32.432020 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:32.431985 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94274df-30a2-4996-920f-f991e35aa6d2-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:57:32.432020 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:32.432013 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njb8p\" (UniqueName: \"kubernetes.io/projected/a94274df-30a2-4996-920f-f991e35aa6d2-kube-api-access-njb8p\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 10 23:57:33.218612 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:33.218583 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" Sep 10 23:57:33.219031 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:33.218615 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5" event={"ID":"a94274df-30a2-4996-920f-f991e35aa6d2","Type":"ContainerDied","Data":"128849c6cc28d7fd188cf9e28da3a9a38004c145eefba4e652e65d90a0ea1ec1"} Sep 10 23:57:33.219031 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:33.218670 2583 scope.go:117] "RemoveContainer" containerID="bae8f3bc1b0fcb63c2977f60df334f89ea4532470c614daaa9fc6799be620bcf" Sep 10 23:57:33.242389 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:33.242362 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5"] Sep 10 23:57:33.246989 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:33.246963 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8d9df10-8dfdd5857-4wcf5"] Sep 10 23:57:33.305452 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:33.305423 2583 scope.go:117] "RemoveContainer" containerID="d020d276c9163c632a0dce2821559b0e69aca4d1556be75fe35c0234b1c99f05" Sep 10 23:57:33.307719 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:33.307694 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="205a63cb-9162-44f5-9f55-91618e82bf17" path="/var/lib/kubelet/pods/205a63cb-9162-44f5-9f55-91618e82bf17/volumes" Sep 10 23:57:33.308149 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:57:33.308131 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a94274df-30a2-4996-920f-f991e35aa6d2" path="/var/lib/kubelet/pods/a94274df-30a2-4996-920f-f991e35aa6d2/volumes" Sep 10 23:58:33.327673 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:58:33.327642 2583 scope.go:117] "RemoveContainer" containerID="982134d65473497d6f79248f31ddf5da9d3df5224d59938765ad8d94671f0f5f" Sep 10 23:58:33.335578 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:58:33.335558 2583 scope.go:117] "RemoveContainer" containerID="2c64d6dff7a79775dcd197541cc63728b0ff46495ebed22b2016a93e96a0abdf" Sep 10 23:59:30.055970 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.055937 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k"] Sep 10 23:59:30.056405 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.056330 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94274df-30a2-4996-920f-f991e35aa6d2" containerName="payload-processing" Sep 10 23:59:30.056405 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.056347 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94274df-30a2-4996-920f-f991e35aa6d2" containerName="payload-processing" Sep 10 23:59:30.056405 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.056363 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="205a63cb-9162-44f5-9f55-91618e82bf17" containerName="revoke-keys" Sep 10 23:59:30.056405 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.056371 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="205a63cb-9162-44f5-9f55-91618e82bf17" containerName="revoke-keys" Sep 10 23:59:30.056541 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.056481 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="205a63cb-9162-44f5-9f55-91618e82bf17" containerName="revoke-keys" Sep 10 23:59:30.056541 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.056496 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94274df-30a2-4996-920f-f991e35aa6d2" containerName="payload-processing" Sep 10 23:59:30.059431 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.059415 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" Sep 10 23:59:30.061626 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.061603 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-afe559-dockercfg-jh5pf\"" Sep 10 23:59:30.061774 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.061754 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-afe559\"" Sep 10 23:59:30.075731 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.075707 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k"] Sep 10 23:59:30.143611 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.143569 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbvqc\" (UniqueName: \"kubernetes.io/projected/44018d2a-70e8-4d43-8a53-9ec08cca4cd2-kube-api-access-nbvqc\") pod \"payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k\" (UID: \"44018d2a-70e8-4d43-8a53-9ec08cca4cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" Sep 10 23:59:30.143804 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.143631 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44018d2a-70e8-4d43-8a53-9ec08cca4cd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k\" (UID: \"44018d2a-70e8-4d43-8a53-9ec08cca4cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" Sep 10 23:59:30.244649 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.244611 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbvqc\" (UniqueName: \"kubernetes.io/projected/44018d2a-70e8-4d43-8a53-9ec08cca4cd2-kube-api-access-nbvqc\") pod \"payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k\" (UID: \"44018d2a-70e8-4d43-8a53-9ec08cca4cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" Sep 10 23:59:30.244847 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.244660 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44018d2a-70e8-4d43-8a53-9ec08cca4cd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k\" (UID: \"44018d2a-70e8-4d43-8a53-9ec08cca4cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" Sep 10 23:59:30.245729 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.245699 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44018d2a-70e8-4d43-8a53-9ec08cca4cd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k\" (UID: \"44018d2a-70e8-4d43-8a53-9ec08cca4cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" Sep 10 23:59:30.252602 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.252580 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbvqc\" (UniqueName: \"kubernetes.io/projected/44018d2a-70e8-4d43-8a53-9ec08cca4cd2-kube-api-access-nbvqc\") pod \"payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k\" (UID: \"44018d2a-70e8-4d43-8a53-9ec08cca4cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" Sep 10 23:59:30.369600 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.369572 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" Sep 10 23:59:30.639403 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.639377 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k"] Sep 10 23:59:30.641372 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:59:30.641346 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44018d2a_70e8_4d43_8a53_9ec08cca4cd2.slice/crio-54c8f61fe2fa78284c77abdf5fcf2cd14473d2d2b8b498fd2287d5bc5221b151 WatchSource:0}: Error finding container 54c8f61fe2fa78284c77abdf5fcf2cd14473d2d2b8b498fd2287d5bc5221b151: Status 404 returned error can't find the container with id 54c8f61fe2fa78284c77abdf5fcf2cd14473d2d2b8b498fd2287d5bc5221b151 Sep 10 23:59:30.809149 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.809105 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" event={"ID":"44018d2a-70e8-4d43-8a53-9ec08cca4cd2","Type":"ContainerStarted","Data":"ef1ae68b111617c1cbaa3be1cb80946f721c0d85b4d88591110a44cae10d2fcf"} Sep 10 23:59:30.809334 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.809170 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" event={"ID":"44018d2a-70e8-4d43-8a53-9ec08cca4cd2","Type":"ContainerStarted","Data":"54c8f61fe2fa78284c77abdf5fcf2cd14473d2d2b8b498fd2287d5bc5221b151"} Sep 10 23:59:30.809334 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.809318 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" Sep 10 23:59:30.824968 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:30.824921 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" podStartSLOduration=0.824906734 podStartE2EDuration="824.906734ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:59:30.823313228 +0000 UTC m=+1378.125791242" watchObservedRunningTime="2026-09-10 23:59:30.824906734 +0000 UTC m=+1378.127384745" Sep 10 23:59:33.356345 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:33.356313 2583 scope.go:117] "RemoveContainer" containerID="2e1f3bcadd655f38dc8632a46b29dda579c0ec071a9ba9140ea5a2ad04728591" Sep 10 23:59:41.816615 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:41.816585 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" Sep 10 23:59:57.017935 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.017900 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw"] Sep 10 23:59:57.022820 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.022797 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.028131 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.028111 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-02bac9-openshift-default-dockercfg-hwndq\"" Sep 10 23:59:57.049567 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.049539 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw"] Sep 10 23:59:57.073900 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.073875 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-workload-socket\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.074035 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.073905 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwtgs\" (UniqueName: \"kubernetes.io/projected/f7a78418-f2e2-4235-9948-40c4cd979c8c-kube-api-access-xwtgs\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.074035 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.073927 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-envoy\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.074035 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.073960 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-data\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.074035 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.074028 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-token\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.074230 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.074062 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-credential-socket\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.074230 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.074108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-podinfo\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.074298 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.074226 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-workload-certs\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.074298 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.074255 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a78418-f2e2-4235-9948-40c4cd979c8c-istiod-ca-cert\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.175200 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.175165 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-credential-socket\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.175200 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.175203 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-podinfo\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.175407 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.175340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-workload-certs\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.175407 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.175386 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a78418-f2e2-4235-9948-40c4cd979c8c-istiod-ca-cert\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.175499 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.175467 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-workload-socket\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.175499 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.175492 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwtgs\" (UniqueName: \"kubernetes.io/projected/f7a78418-f2e2-4235-9948-40c4cd979c8c-kube-api-access-xwtgs\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.175603 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.175521 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-envoy\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.175603 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.175550 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-data\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.175715 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.175610 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-token\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.175715 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.175625 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-credential-socket\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.175715 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.175668 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-workload-certs\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.175875 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.175802 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-workload-socket\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.175957 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.175934 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-data\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.176209 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.176187 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a78418-f2e2-4235-9948-40c4cd979c8c-istiod-ca-cert\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.177724 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.177703 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-envoy\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.177856 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.177839 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-podinfo\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.183904 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.183883 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-token\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.184474 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.184450 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwtgs\" (UniqueName: \"kubernetes.io/projected/f7a78418-f2e2-4235-9948-40c4cd979c8c-kube-api-access-xwtgs\") pod \"e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.332833 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.332756 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:57.611621 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.611578 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw"] Sep 10 23:59:57.615311 ip-10-0-134-90 kubenswrapper[2583]: W0910 23:59:57.615280 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a78418_f2e2_4235_9948_40c4cd979c8c.slice/crio-c3dc59b40d9f0147d3e1ef4ae44348f262f477d0be423d0da46d3d7943f786dd WatchSource:0}: Error finding container c3dc59b40d9f0147d3e1ef4ae44348f262f477d0be423d0da46d3d7943f786dd: Status 404 returned error can't find the container with id c3dc59b40d9f0147d3e1ef4ae44348f262f477d0be423d0da46d3d7943f786dd Sep 10 23:59:57.618000 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.617966 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:59:57.618136 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.618043 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:59:57.618136 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.618090 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:59:57.949856 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.949763 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" event={"ID":"f7a78418-f2e2-4235-9948-40c4cd979c8c","Type":"ContainerStarted","Data":"6aa1c7bbd33a6f7c9a71d3eb9300bd191b72ad12b729857fc062088eaf2c56b5"} Sep 10 23:59:57.950003 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.949859 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" event={"ID":"f7a78418-f2e2-4235-9948-40c4cd979c8c","Type":"ContainerStarted","Data":"c3dc59b40d9f0147d3e1ef4ae44348f262f477d0be423d0da46d3d7943f786dd"} Sep 10 23:59:57.970582 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:57.970526 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" podStartSLOduration=1.970512525 podStartE2EDuration="1.970512525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:59:57.96850844 +0000 UTC m=+1405.270986460" watchObservedRunningTime="2026-09-10 23:59:57.970512525 +0000 UTC m=+1405.272990535" Sep 10 23:59:58.333655 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:58.333567 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:58.338558 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:58.338532 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:58.954949 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:58.954911 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 10 23:59:58.955779 ip-10-0-134-90 kubenswrapper[2583]: I0910 23:59:58.955757 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 11 00:00:00.169102 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.169058 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj"] Sep 11 00:00:00.172535 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.172518 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj" Sep 11 00:00:00.175915 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.175892 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 11 00:00:00.188117 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.188094 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818080-g8d78"] Sep 11 00:00:00.191968 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.191697 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5"] Sep 11 00:00:00.193371 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.192881 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-g8d78" Sep 11 00:00:00.195621 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.195603 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj"] Sep 11 00:00:00.195867 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.195749 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5" Sep 11 00:00:00.205542 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.205521 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj" Sep 11 00:00:00.218090 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.218053 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-g8d78" Sep 11 00:00:00.221530 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.221505 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5"] Sep 11 00:00:00.234920 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.234892 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818080-g8d78"] Sep 11 00:00:00.260641 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.260431 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5" Sep 11 00:00:00.506258 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.506231 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj"] Sep 11 00:00:00.508295 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:00:00.508266 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d858ad_b5ef_4390_a81b_2459f1dd367f.slice/crio-95be38311e873e876e2661fc49460e8a952eed42583609f737ff281b0288ce1c WatchSource:0}: Error finding container 95be38311e873e876e2661fc49460e8a952eed42583609f737ff281b0288ce1c: Status 404 returned error can't find the container with id 95be38311e873e876e2661fc49460e8a952eed42583609f737ff281b0288ce1c Sep 11 00:00:00.526749 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.526730 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818080-g8d78"] Sep 11 00:00:00.528263 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:00:00.528235 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129166ea_7e5c_4713_832d_53942e7b36f7.slice/crio-1a27e741f96784bc81014e47226fbc74efe8d18fce3376d507f06adbea036165 WatchSource:0}: Error finding container 1a27e741f96784bc81014e47226fbc74efe8d18fce3376d507f06adbea036165: Status 404 returned error can't find the container with id 1a27e741f96784bc81014e47226fbc74efe8d18fce3376d507f06adbea036165 Sep 11 00:00:00.559698 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.559673 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5"] Sep 11 00:00:00.561770 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:00:00.561751 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccca1f6_83a9_4bdf_8f50_67ad74569188.slice/crio-0553b3321104602b2552e3d805db17ac1dee6db9c9668f3d745cea5bab4c1f88 WatchSource:0}: Error finding container 0553b3321104602b2552e3d805db17ac1dee6db9c9668f3d745cea5bab4c1f88: Status 404 returned error can't find the container with id 0553b3321104602b2552e3d805db17ac1dee6db9c9668f3d745cea5bab4c1f88 Sep 11 00:00:00.966375 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.966339 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5" event={"ID":"accca1f6-83a9-4bdf-8f50-67ad74569188","Type":"ContainerStarted","Data":"9b01e557f346bd9bba459511c6254bed28a66b16d0e17a0b9f1f38fddac74ece"} Sep 11 00:00:00.966546 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.966396 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5" event={"ID":"accca1f6-83a9-4bdf-8f50-67ad74569188","Type":"ContainerStarted","Data":"0553b3321104602b2552e3d805db17ac1dee6db9c9668f3d745cea5bab4c1f88"} Sep 11 00:00:00.968016 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.967986 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj" event={"ID":"e1d858ad-b5ef-4390-a81b-2459f1dd367f","Type":"ContainerStarted","Data":"b2af44b19a5b0861e693383670efd456be1eb36080ddb0f6ba16d874abdadbed"} Sep 11 00:00:00.968173 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.968036 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj" event={"ID":"e1d858ad-b5ef-4390-a81b-2459f1dd367f","Type":"ContainerStarted","Data":"95be38311e873e876e2661fc49460e8a952eed42583609f737ff281b0288ce1c"} Sep 11 00:00:00.969495 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.969475 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-g8d78" event={"ID":"129166ea-7e5c-4713-832d-53942e7b36f7","Type":"ContainerStarted","Data":"d33ab4410628db26f7202c7b7c40f0886341bae7dc3789592bcc65faa502530e"} Sep 11 00:00:00.969587 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.969511 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-g8d78" event={"ID":"129166ea-7e5c-4713-832d-53942e7b36f7","Type":"ContainerStarted","Data":"1a27e741f96784bc81014e47226fbc74efe8d18fce3376d507f06adbea036165"} Sep 11 00:00:00.984054 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.984011 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5" podStartSLOduration=0.983997934 podStartE2EDuration="983.997934ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:00.981827123 +0000 UTC m=+1408.284305135" watchObservedRunningTime="2026-09-11 00:00:00.983997934 +0000 UTC m=+1408.286475946" Sep 11 00:00:00.998466 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:00.998425 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj" podStartSLOduration=0.998411199 podStartE2EDuration="998.411199ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:00.996165804 +0000 UTC m=+1408.298643818" watchObservedRunningTime="2026-09-11 00:00:00.998411199 +0000 UTC m=+1408.300889210" Sep 11 00:00:01.015856 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:01.015818 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-g8d78" podStartSLOduration=1.015806128 podStartE2EDuration="1.015806128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:01.015212417 +0000 UTC m=+1408.317690425" watchObservedRunningTime="2026-09-11 00:00:01.015806128 +0000 UTC m=+1408.318284156" Sep 11 00:00:01.976916 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:01.976874 2583 generic.go:356] "Generic (PLEG): container finished" podID="129166ea-7e5c-4713-832d-53942e7b36f7" containerID="d33ab4410628db26f7202c7b7c40f0886341bae7dc3789592bcc65faa502530e" exitCode=0 Sep 11 00:00:01.977336 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:01.976962 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-g8d78" event={"ID":"129166ea-7e5c-4713-832d-53942e7b36f7","Type":"ContainerDied","Data":"d33ab4410628db26f7202c7b7c40f0886341bae7dc3789592bcc65faa502530e"} Sep 11 00:00:01.978788 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:01.978717 2583 generic.go:356] "Generic (PLEG): container finished" podID="accca1f6-83a9-4bdf-8f50-67ad74569188" containerID="9b01e557f346bd9bba459511c6254bed28a66b16d0e17a0b9f1f38fddac74ece" exitCode=0 Sep 11 00:00:01.978788 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:01.978757 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5" event={"ID":"accca1f6-83a9-4bdf-8f50-67ad74569188","Type":"ContainerDied","Data":"9b01e557f346bd9bba459511c6254bed28a66b16d0e17a0b9f1f38fddac74ece"} Sep 11 00:00:01.980354 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:01.980331 2583 generic.go:356] "Generic (PLEG): container finished" podID="e1d858ad-b5ef-4390-a81b-2459f1dd367f" containerID="b2af44b19a5b0861e693383670efd456be1eb36080ddb0f6ba16d874abdadbed" exitCode=0 Sep 11 00:00:01.980487 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:01.980362 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj" event={"ID":"e1d858ad-b5ef-4390-a81b-2459f1dd367f","Type":"ContainerDied","Data":"b2af44b19a5b0861e693383670efd456be1eb36080ddb0f6ba16d874abdadbed"} Sep 11 00:00:03.122868 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:03.122831 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-g8d78" Sep 11 00:00:03.169776 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:03.169742 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj" Sep 11 00:00:03.172910 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:03.172891 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5" Sep 11 00:00:03.992596 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:03.992565 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5" Sep 11 00:00:03.992804 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:03.992565 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5" event={"ID":"accca1f6-83a9-4bdf-8f50-67ad74569188","Type":"ContainerDied","Data":"0553b3321104602b2552e3d805db17ac1dee6db9c9668f3d745cea5bab4c1f88"} Sep 11 00:00:03.992804 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:03.992700 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0553b3321104602b2552e3d805db17ac1dee6db9c9668f3d745cea5bab4c1f88" Sep 11 00:00:03.993957 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:03.993939 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj" Sep 11 00:00:03.994106 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:03.993983 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj" event={"ID":"e1d858ad-b5ef-4390-a81b-2459f1dd367f","Type":"ContainerDied","Data":"95be38311e873e876e2661fc49460e8a952eed42583609f737ff281b0288ce1c"} Sep 11 00:00:03.994106 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:03.994023 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95be38311e873e876e2661fc49460e8a952eed42583609f737ff281b0288ce1c" Sep 11 00:00:03.995422 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:03.995403 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-g8d78" event={"ID":"129166ea-7e5c-4713-832d-53942e7b36f7","Type":"ContainerDied","Data":"1a27e741f96784bc81014e47226fbc74efe8d18fce3376d507f06adbea036165"} Sep 11 00:00:03.995518 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:03.995435 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a27e741f96784bc81014e47226fbc74efe8d18fce3376d507f06adbea036165" Sep 11 00:00:03.995518 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:03.995439 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-g8d78" Sep 11 00:00:22.092317 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:22.092284 2583 generic.go:356] "Generic (PLEG): container finished" podID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerID="6aa1c7bbd33a6f7c9a71d3eb9300bd191b72ad12b729857fc062088eaf2c56b5" exitCode=0 Sep 11 00:00:22.092722 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:22.092322 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" event={"ID":"f7a78418-f2e2-4235-9948-40c4cd979c8c","Type":"ContainerDied","Data":"6aa1c7bbd33a6f7c9a71d3eb9300bd191b72ad12b729857fc062088eaf2c56b5"} Sep 11 00:00:22.092722 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:22.092694 2583 scope.go:117] "RemoveContainer" containerID="6aa1c7bbd33a6f7c9a71d3eb9300bd191b72ad12b729857fc062088eaf2c56b5" Sep 11 00:00:22.093756 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:22.093732 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:00:22.093812 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:22.093795 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:00:22.093848 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:22.093824 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:00:23.099670 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:23.099637 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" event={"ID":"f7a78418-f2e2-4235-9948-40c4cd979c8c","Type":"ContainerStarted","Data":"f1374736f51931bd80e29c1227d64e9197d1b74666dffbbd00cc2a082864911c"} Sep 11 00:00:23.333359 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:23.333317 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 11 00:00:23.333589 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:23.333446 2583 patch_prober.go:28] interesting pod/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 11 00:00:23.333589 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:23.333508 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 11 00:00:24.334220 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:24.334174 2583 patch_prober.go:28] interesting pod/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 11 00:00:24.334733 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:24.334245 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 11 00:00:25.342604 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:25.342559 2583 patch_prober.go:28] interesting pod/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 11 00:00:25.343141 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:25.342631 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 11 00:00:26.333236 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:26.333198 2583 patch_prober.go:28] interesting pod/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 11 00:00:26.333553 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:26.333522 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 11 00:00:27.334209 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:27.334167 2583 patch_prober.go:28] interesting pod/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 11 00:00:27.334692 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:27.334233 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 11 00:00:28.333666 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:28.333612 2583 patch_prober.go:28] interesting pod/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 11 00:00:28.333868 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:28.333725 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 11 00:00:28.955329 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:28.955278 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 11 00:00:29.337853 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:29.337768 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 11 00:00:29.338761 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:29.338740 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 11 00:00:32.883459 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.883425 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh"] Sep 11 00:00:32.883821 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.883780 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1d858ad-b5ef-4390-a81b-2459f1dd367f" containerName="cleanup" Sep 11 00:00:32.883821 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.883791 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1d858ad-b5ef-4390-a81b-2459f1dd367f" containerName="cleanup" Sep 11 00:00:32.883821 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.883808 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="accca1f6-83a9-4bdf-8f50-67ad74569188" containerName="cleanup" Sep 11 00:00:32.883821 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.883813 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="accca1f6-83a9-4bdf-8f50-67ad74569188" containerName="cleanup" Sep 11 00:00:32.883821 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.883821 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="129166ea-7e5c-4713-832d-53942e7b36f7" containerName="cleanup" Sep 11 00:00:32.883984 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.883827 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="129166ea-7e5c-4713-832d-53942e7b36f7" containerName="cleanup" Sep 11 00:00:32.883984 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.883880 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1d858ad-b5ef-4390-a81b-2459f1dd367f" containerName="cleanup" Sep 11 00:00:32.883984 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.883889 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="129166ea-7e5c-4713-832d-53942e7b36f7" containerName="cleanup" Sep 11 00:00:32.883984 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.883894 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="accca1f6-83a9-4bdf-8f50-67ad74569188" containerName="cleanup" Sep 11 00:00:32.888025 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.888008 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh" Sep 11 00:00:32.891707 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.891672 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:00:32.891838 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.891775 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 11 00:00:32.900273 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:32.900247 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh"] Sep 11 00:00:33.019041 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:33.019010 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8765ffcc-d6cb-4a09-8daa-7d8916cf3922-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-afe559-j86nh\" (UID: \"8765ffcc-d6cb-4a09-8daa-7d8916cf3922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh" Sep 11 00:00:33.120335 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:33.120302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8765ffcc-d6cb-4a09-8daa-7d8916cf3922-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-afe559-j86nh\" (UID: \"8765ffcc-d6cb-4a09-8daa-7d8916cf3922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh" Sep 11 00:00:33.122549 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:33.122519 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:00:33.130976 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:33.130954 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8765ffcc-d6cb-4a09-8daa-7d8916cf3922-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-afe559-j86nh\" (UID: \"8765ffcc-d6cb-4a09-8daa-7d8916cf3922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh" Sep 11 00:00:33.201021 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:33.200943 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 11 00:00:33.208752 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:33.208723 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh" Sep 11 00:00:33.477225 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:33.477200 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh"] Sep 11 00:00:33.478720 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:00:33.478691 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8765ffcc_d6cb_4a09_8daa_7d8916cf3922.slice/crio-95c838074614cbdd7072b741b2c9dc33af3d085c8aedb19075b9975481f89636 WatchSource:0}: Error finding container 95c838074614cbdd7072b741b2c9dc33af3d085c8aedb19075b9975481f89636: Status 404 returned error can't find the container with id 95c838074614cbdd7072b741b2c9dc33af3d085c8aedb19075b9975481f89636 Sep 11 00:00:34.162616 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:34.162586 2583 generic.go:356] "Generic (PLEG): container finished" podID="8765ffcc-d6cb-4a09-8daa-7d8916cf3922" containerID="290725d5e5e2c3386b77456b50740a4237d98363eed9c356c967c397c3cc0c45" exitCode=0 Sep 11 00:00:34.162962 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:34.162674 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh" event={"ID":"8765ffcc-d6cb-4a09-8daa-7d8916cf3922","Type":"ContainerDied","Data":"290725d5e5e2c3386b77456b50740a4237d98363eed9c356c967c397c3cc0c45"} Sep 11 00:00:34.162962 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:34.162726 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh" event={"ID":"8765ffcc-d6cb-4a09-8daa-7d8916cf3922","Type":"ContainerStarted","Data":"95c838074614cbdd7072b741b2c9dc33af3d085c8aedb19075b9975481f89636"} Sep 11 00:00:35.305205 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:35.305179 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh" Sep 11 00:00:35.346220 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:35.340296 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8765ffcc-d6cb-4a09-8daa-7d8916cf3922-maas-api-service-ca\") pod \"8765ffcc-d6cb-4a09-8daa-7d8916cf3922\" (UID: \"8765ffcc-d6cb-4a09-8daa-7d8916cf3922\") " Sep 11 00:00:35.346220 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:35.341000 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8765ffcc-d6cb-4a09-8daa-7d8916cf3922-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8765ffcc-d6cb-4a09-8daa-7d8916cf3922" (UID: "8765ffcc-d6cb-4a09-8daa-7d8916cf3922"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:00:35.441597 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:35.441521 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8765ffcc-d6cb-4a09-8daa-7d8916cf3922-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:00:36.175943 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:36.175901 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh" event={"ID":"8765ffcc-d6cb-4a09-8daa-7d8916cf3922","Type":"ContainerDied","Data":"95c838074614cbdd7072b741b2c9dc33af3d085c8aedb19075b9975481f89636"} Sep 11 00:00:36.176109 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:36.175982 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95c838074614cbdd7072b741b2c9dc33af3d085c8aedb19075b9975481f89636" Sep 11 00:00:36.176109 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:36.175911 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh" Sep 11 00:00:48.627930 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:48.627894 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k"] Sep 11 00:00:48.628415 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:48.628181 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" podUID="44018d2a-70e8-4d43-8a53-9ec08cca4cd2" containerName="payload-pre-processing" containerID="cri-o://ef1ae68b111617c1cbaa3be1cb80946f721c0d85b4d88591110a44cae10d2fcf" gracePeriod=30 Sep 11 00:00:48.799632 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:48.799570 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh"] Sep 11 00:00:48.801758 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:48.801733 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-afe559-j86nh"] Sep 11 00:00:48.895360 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:48.895010 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" Sep 11 00:00:48.959315 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:48.959287 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbvqc\" (UniqueName: \"kubernetes.io/projected/44018d2a-70e8-4d43-8a53-9ec08cca4cd2-kube-api-access-nbvqc\") pod \"44018d2a-70e8-4d43-8a53-9ec08cca4cd2\" (UID: \"44018d2a-70e8-4d43-8a53-9ec08cca4cd2\") " Sep 11 00:00:48.959494 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:48.959331 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44018d2a-70e8-4d43-8a53-9ec08cca4cd2-plugins-config-volume\") pod \"44018d2a-70e8-4d43-8a53-9ec08cca4cd2\" (UID: \"44018d2a-70e8-4d43-8a53-9ec08cca4cd2\") " Sep 11 00:00:48.959730 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:48.959707 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44018d2a-70e8-4d43-8a53-9ec08cca4cd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44018d2a-70e8-4d43-8a53-9ec08cca4cd2" (UID: "44018d2a-70e8-4d43-8a53-9ec08cca4cd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:00:48.961449 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:48.961428 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44018d2a-70e8-4d43-8a53-9ec08cca4cd2-kube-api-access-nbvqc" (OuterVolumeSpecName: "kube-api-access-nbvqc") pod "44018d2a-70e8-4d43-8a53-9ec08cca4cd2" (UID: "44018d2a-70e8-4d43-8a53-9ec08cca4cd2"). InnerVolumeSpecName "kube-api-access-nbvqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:00:49.060145 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:49.060114 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbvqc\" (UniqueName: \"kubernetes.io/projected/44018d2a-70e8-4d43-8a53-9ec08cca4cd2-kube-api-access-nbvqc\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:00:49.060145 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:49.060140 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44018d2a-70e8-4d43-8a53-9ec08cca4cd2-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:00:49.242248 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:49.242156 2583 generic.go:356] "Generic (PLEG): container finished" podID="44018d2a-70e8-4d43-8a53-9ec08cca4cd2" containerID="ef1ae68b111617c1cbaa3be1cb80946f721c0d85b4d88591110a44cae10d2fcf" exitCode=0 Sep 11 00:00:49.242248 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:49.242203 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" Sep 11 00:00:49.242455 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:49.242248 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" event={"ID":"44018d2a-70e8-4d43-8a53-9ec08cca4cd2","Type":"ContainerDied","Data":"ef1ae68b111617c1cbaa3be1cb80946f721c0d85b4d88591110a44cae10d2fcf"} Sep 11 00:00:49.242455 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:49.242300 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k" event={"ID":"44018d2a-70e8-4d43-8a53-9ec08cca4cd2","Type":"ContainerDied","Data":"54c8f61fe2fa78284c77abdf5fcf2cd14473d2d2b8b498fd2287d5bc5221b151"} Sep 11 00:00:49.242455 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:49.242323 2583 scope.go:117] "RemoveContainer" containerID="ef1ae68b111617c1cbaa3be1cb80946f721c0d85b4d88591110a44cae10d2fcf" Sep 11 00:00:49.252356 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:49.252333 2583 scope.go:117] "RemoveContainer" containerID="ef1ae68b111617c1cbaa3be1cb80946f721c0d85b4d88591110a44cae10d2fcf" Sep 11 00:00:49.252635 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:00:49.252615 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1ae68b111617c1cbaa3be1cb80946f721c0d85b4d88591110a44cae10d2fcf\": container with ID starting with ef1ae68b111617c1cbaa3be1cb80946f721c0d85b4d88591110a44cae10d2fcf not found: ID does not exist" containerID="ef1ae68b111617c1cbaa3be1cb80946f721c0d85b4d88591110a44cae10d2fcf" Sep 11 00:00:49.252722 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:49.252644 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1ae68b111617c1cbaa3be1cb80946f721c0d85b4d88591110a44cae10d2fcf"} err="failed to get container status \"ef1ae68b111617c1cbaa3be1cb80946f721c0d85b4d88591110a44cae10d2fcf\": rpc error: code = NotFound desc = could not find container \"ef1ae68b111617c1cbaa3be1cb80946f721c0d85b4d88591110a44cae10d2fcf\": container with ID starting with ef1ae68b111617c1cbaa3be1cb80946f721c0d85b4d88591110a44cae10d2fcf not found: ID does not exist" Sep 11 00:00:49.268245 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:49.268221 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k"] Sep 11 00:00:49.273235 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:49.273208 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-afe559-7bcd688c66-k6z5k"] Sep 11 00:00:49.307988 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:49.307958 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44018d2a-70e8-4d43-8a53-9ec08cca4cd2" path="/var/lib/kubelet/pods/44018d2a-70e8-4d43-8a53-9ec08cca4cd2/volumes" Sep 11 00:00:49.308320 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:49.308307 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8765ffcc-d6cb-4a09-8daa-7d8916cf3922" path="/var/lib/kubelet/pods/8765ffcc-d6cb-4a09-8daa-7d8916cf3922/volumes" Sep 11 00:00:55.496308 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.496269 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z"] Sep 11 00:00:55.496681 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.496603 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8765ffcc-d6cb-4a09-8daa-7d8916cf3922" containerName="revoke-keys" Sep 11 00:00:55.496681 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.496615 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8765ffcc-d6cb-4a09-8daa-7d8916cf3922" containerName="revoke-keys" Sep 11 00:00:55.496681 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.496637 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44018d2a-70e8-4d43-8a53-9ec08cca4cd2" containerName="payload-pre-processing" Sep 11 00:00:55.496681 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.496643 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="44018d2a-70e8-4d43-8a53-9ec08cca4cd2" containerName="payload-pre-processing" Sep 11 00:00:55.496821 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.496699 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8765ffcc-d6cb-4a09-8daa-7d8916cf3922" containerName="revoke-keys" Sep 11 00:00:55.496821 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.496707 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="44018d2a-70e8-4d43-8a53-9ec08cca4cd2" containerName="payload-pre-processing" Sep 11 00:00:55.501253 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.501233 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z" Sep 11 00:00:55.504028 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.504007 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 11 00:00:55.504251 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.504234 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:00:55.508383 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.508037 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z"] Sep 11 00:00:55.618484 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.618448 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2ee3c8-b43a-4922-8aed-d00f0c715df4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z\" (UID: \"6f2ee3c8-b43a-4922-8aed-d00f0c715df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z" Sep 11 00:00:55.719671 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.719622 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2ee3c8-b43a-4922-8aed-d00f0c715df4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z\" (UID: \"6f2ee3c8-b43a-4922-8aed-d00f0c715df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z" Sep 11 00:00:55.720545 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.720520 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2ee3c8-b43a-4922-8aed-d00f0c715df4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z\" (UID: \"6f2ee3c8-b43a-4922-8aed-d00f0c715df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z" Sep 11 00:00:55.811743 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:55.811659 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z" Sep 11 00:00:56.082279 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:56.082252 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z"] Sep 11 00:00:56.084181 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:00:56.084150 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2ee3c8_b43a_4922_8aed_d00f0c715df4.slice/crio-ade89eaf80b476c5c5a89541f244e20d10118a35d4a4915fa01103ad29fb9c0d WatchSource:0}: Error finding container ade89eaf80b476c5c5a89541f244e20d10118a35d4a4915fa01103ad29fb9c0d: Status 404 returned error can't find the container with id ade89eaf80b476c5c5a89541f244e20d10118a35d4a4915fa01103ad29fb9c0d Sep 11 00:00:56.287703 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:56.287670 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z" event={"ID":"6f2ee3c8-b43a-4922-8aed-d00f0c715df4","Type":"ContainerStarted","Data":"dcedd1b5e3cb27e24217458c38818088b11b96a2bfb3326fa351238ce01f55e9"} Sep 11 00:00:56.287831 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:56.287723 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z" event={"ID":"6f2ee3c8-b43a-4922-8aed-d00f0c715df4","Type":"ContainerStarted","Data":"ade89eaf80b476c5c5a89541f244e20d10118a35d4a4915fa01103ad29fb9c0d"} Sep 11 00:00:56.304231 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:56.304178 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z" podStartSLOduration=1.3041618 podStartE2EDuration="1.3041618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:56.302192223 +0000 UTC m=+1463.604670238" watchObservedRunningTime="2026-09-11 00:00:56.3041618 +0000 UTC m=+1463.606639809" Sep 11 00:00:57.296732 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:57.296697 2583 generic.go:356] "Generic (PLEG): container finished" podID="6f2ee3c8-b43a-4922-8aed-d00f0c715df4" containerID="dcedd1b5e3cb27e24217458c38818088b11b96a2bfb3326fa351238ce01f55e9" exitCode=0 Sep 11 00:00:57.297105 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:57.296774 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z" event={"ID":"6f2ee3c8-b43a-4922-8aed-d00f0c715df4","Type":"ContainerDied","Data":"dcedd1b5e3cb27e24217458c38818088b11b96a2bfb3326fa351238ce01f55e9"} Sep 11 00:00:58.435572 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:58.435539 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z" Sep 11 00:00:58.546149 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:58.546113 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2ee3c8-b43a-4922-8aed-d00f0c715df4-maas-api-service-ca\") pod \"6f2ee3c8-b43a-4922-8aed-d00f0c715df4\" (UID: \"6f2ee3c8-b43a-4922-8aed-d00f0c715df4\") " Sep 11 00:00:58.546431 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:58.546409 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2ee3c8-b43a-4922-8aed-d00f0c715df4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f2ee3c8-b43a-4922-8aed-d00f0c715df4" (UID: "6f2ee3c8-b43a-4922-8aed-d00f0c715df4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:00:58.647506 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:58.647474 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2ee3c8-b43a-4922-8aed-d00f0c715df4-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:00:59.308627 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:59.308594 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z" Sep 11 00:00:59.308627 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:59.308601 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z" event={"ID":"6f2ee3c8-b43a-4922-8aed-d00f0c715df4","Type":"ContainerDied","Data":"ade89eaf80b476c5c5a89541f244e20d10118a35d4a4915fa01103ad29fb9c0d"} Sep 11 00:00:59.308835 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:00:59.308643 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ade89eaf80b476c5c5a89541f244e20d10118a35d4a4915fa01103ad29fb9c0d" Sep 11 00:01:11.239231 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:11.239195 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z"] Sep 11 00:01:11.241965 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:11.241937 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02bac9-rhd2z"] Sep 11 00:01:11.308207 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:11.308171 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f2ee3c8-b43a-4922-8aed-d00f0c715df4" path="/var/lib/kubelet/pods/6f2ee3c8-b43a-4922-8aed-d00f0c715df4/volumes" Sep 11 00:01:17.496673 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:17.496639 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw"] Sep 11 00:01:17.497059 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:17.496909 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerName="istio-proxy" containerID="cri-o://f1374736f51931bd80e29c1227d64e9197d1b74666dffbbd00cc2a082864911c" gracePeriod=30 Sep 11 00:01:17.884223 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:17.884193 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn"] Sep 11 00:01:17.884570 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:17.884558 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2ee3c8-b43a-4922-8aed-d00f0c715df4" containerName="revoke-keys" Sep 11 00:01:17.884615 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:17.884572 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2ee3c8-b43a-4922-8aed-d00f0c715df4" containerName="revoke-keys" Sep 11 00:01:17.884651 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:17.884627 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f2ee3c8-b43a-4922-8aed-d00f0c715df4" containerName="revoke-keys" Sep 11 00:01:17.887767 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:17.887752 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn" Sep 11 00:01:17.890247 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:17.890221 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:01:17.890359 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:17.890243 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 11 00:01:17.894632 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:17.894608 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn"] Sep 11 00:01:18.023022 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:18.022987 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/602d8cdf-5ff9-4d49-96b5-af35bc71eb08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn\" (UID: \"602d8cdf-5ff9-4d49-96b5-af35bc71eb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn" Sep 11 00:01:18.123487 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:18.123448 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/602d8cdf-5ff9-4d49-96b5-af35bc71eb08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn\" (UID: \"602d8cdf-5ff9-4d49-96b5-af35bc71eb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn" Sep 11 00:01:18.124099 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:18.124063 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/602d8cdf-5ff9-4d49-96b5-af35bc71eb08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn\" (UID: \"602d8cdf-5ff9-4d49-96b5-af35bc71eb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn" Sep 11 00:01:18.198493 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:18.198419 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn" Sep 11 00:01:18.468106 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:18.468082 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn"] Sep 11 00:01:18.469895 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:01:18.469870 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602d8cdf_5ff9_4d49_96b5_af35bc71eb08.slice/crio-f4fccb93280aace79dd26f773091477100c51be9c6ab5ef30c8e4990b329dea7 WatchSource:0}: Error finding container f4fccb93280aace79dd26f773091477100c51be9c6ab5ef30c8e4990b329dea7: Status 404 returned error can't find the container with id f4fccb93280aace79dd26f773091477100c51be9c6ab5ef30c8e4990b329dea7 Sep 11 00:01:19.416250 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:19.416217 2583 generic.go:356] "Generic (PLEG): container finished" podID="602d8cdf-5ff9-4d49-96b5-af35bc71eb08" containerID="a0990b3d9298b745101c2302c6fa80ec46d601b2c81c7e622a49469f65bb3466" exitCode=0 Sep 11 00:01:19.416610 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:19.416304 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn" event={"ID":"602d8cdf-5ff9-4d49-96b5-af35bc71eb08","Type":"ContainerDied","Data":"a0990b3d9298b745101c2302c6fa80ec46d601b2c81c7e622a49469f65bb3466"} Sep 11 00:01:19.416610 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:19.416360 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn" event={"ID":"602d8cdf-5ff9-4d49-96b5-af35bc71eb08","Type":"ContainerStarted","Data":"f4fccb93280aace79dd26f773091477100c51be9c6ab5ef30c8e4990b329dea7"} Sep 11 00:01:20.551747 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:20.551723 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn" Sep 11 00:01:20.650263 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:20.650226 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/602d8cdf-5ff9-4d49-96b5-af35bc71eb08-maas-api-service-ca\") pod \"602d8cdf-5ff9-4d49-96b5-af35bc71eb08\" (UID: \"602d8cdf-5ff9-4d49-96b5-af35bc71eb08\") " Sep 11 00:01:20.650553 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:20.650530 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602d8cdf-5ff9-4d49-96b5-af35bc71eb08-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "602d8cdf-5ff9-4d49-96b5-af35bc71eb08" (UID: "602d8cdf-5ff9-4d49-96b5-af35bc71eb08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:01:20.750995 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:20.750913 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/602d8cdf-5ff9-4d49-96b5-af35bc71eb08-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:21.427916 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:21.427887 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn" Sep 11 00:01:21.428127 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:21.427928 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn" event={"ID":"602d8cdf-5ff9-4d49-96b5-af35bc71eb08","Type":"ContainerDied","Data":"f4fccb93280aace79dd26f773091477100c51be9c6ab5ef30c8e4990b329dea7"} Sep 11 00:01:21.428127 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:21.427972 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4fccb93280aace79dd26f773091477100c51be9c6ab5ef30c8e4990b329dea7" Sep 11 00:01:22.748595 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.748563 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 11 00:01:22.868019 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.867983 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-data\") pod \"f7a78418-f2e2-4235-9948-40c4cd979c8c\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " Sep 11 00:01:22.868219 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868044 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-podinfo\") pod \"f7a78418-f2e2-4235-9948-40c4cd979c8c\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " Sep 11 00:01:22.868219 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868111 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-credential-socket\") pod \"f7a78418-f2e2-4235-9948-40c4cd979c8c\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " Sep 11 00:01:22.868219 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868141 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-token\") pod \"f7a78418-f2e2-4235-9948-40c4cd979c8c\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " Sep 11 00:01:22.868219 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868188 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-data" (OuterVolumeSpecName: "istio-data") pod "f7a78418-f2e2-4235-9948-40c4cd979c8c" (UID: "f7a78418-f2e2-4235-9948-40c4cd979c8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:22.868219 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868197 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a78418-f2e2-4235-9948-40c4cd979c8c-istiod-ca-cert\") pod \"f7a78418-f2e2-4235-9948-40c4cd979c8c\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " Sep 11 00:01:22.868494 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868250 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-workload-certs\") pod \"f7a78418-f2e2-4235-9948-40c4cd979c8c\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " Sep 11 00:01:22.868494 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868273 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-workload-socket\") pod \"f7a78418-f2e2-4235-9948-40c4cd979c8c\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " Sep 11 00:01:22.868494 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868309 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-envoy\") pod \"f7a78418-f2e2-4235-9948-40c4cd979c8c\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " Sep 11 00:01:22.868494 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868352 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwtgs\" (UniqueName: \"kubernetes.io/projected/f7a78418-f2e2-4235-9948-40c4cd979c8c-kube-api-access-xwtgs\") pod \"f7a78418-f2e2-4235-9948-40c4cd979c8c\" (UID: \"f7a78418-f2e2-4235-9948-40c4cd979c8c\") " Sep 11 00:01:22.868710 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868468 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f7a78418-f2e2-4235-9948-40c4cd979c8c" (UID: "f7a78418-f2e2-4235-9948-40c4cd979c8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:22.868710 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868511 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f7a78418-f2e2-4235-9948-40c4cd979c8c" (UID: "f7a78418-f2e2-4235-9948-40c4cd979c8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:22.868710 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868562 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a78418-f2e2-4235-9948-40c4cd979c8c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f7a78418-f2e2-4235-9948-40c4cd979c8c" (UID: "f7a78418-f2e2-4235-9948-40c4cd979c8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:01:22.868710 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868558 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f7a78418-f2e2-4235-9948-40c4cd979c8c" (UID: "f7a78418-f2e2-4235-9948-40c4cd979c8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:22.868925 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868730 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a78418-f2e2-4235-9948-40c4cd979c8c-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:22.868925 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868751 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:22.868925 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868764 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:22.868925 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868776 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:22.868925 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.868787 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:22.870414 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.870379 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7a78418-f2e2-4235-9948-40c4cd979c8c" (UID: "f7a78418-f2e2-4235-9948-40c4cd979c8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:01:22.870552 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.870426 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f7a78418-f2e2-4235-9948-40c4cd979c8c" (UID: "f7a78418-f2e2-4235-9948-40c4cd979c8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:22.870552 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.870445 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a78418-f2e2-4235-9948-40c4cd979c8c-kube-api-access-xwtgs" (OuterVolumeSpecName: "kube-api-access-xwtgs") pod "f7a78418-f2e2-4235-9948-40c4cd979c8c" (UID: "f7a78418-f2e2-4235-9948-40c4cd979c8c"). InnerVolumeSpecName "kube-api-access-xwtgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:01:22.870552 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.870426 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f7a78418-f2e2-4235-9948-40c4cd979c8c" (UID: "f7a78418-f2e2-4235-9948-40c4cd979c8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:01:22.969900 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.969866 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:22.969900 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.969892 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwtgs\" (UniqueName: \"kubernetes.io/projected/f7a78418-f2e2-4235-9948-40c4cd979c8c-kube-api-access-xwtgs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:22.969900 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.969903 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:22.970156 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:22.969912 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a78418-f2e2-4235-9948-40c4cd979c8c-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:23.440124 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:23.440063 2583 generic.go:356] "Generic (PLEG): container finished" podID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerID="f1374736f51931bd80e29c1227d64e9197d1b74666dffbbd00cc2a082864911c" exitCode=0 Sep 11 00:01:23.440288 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:23.440131 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" Sep 11 00:01:23.440288 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:23.440171 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" event={"ID":"f7a78418-f2e2-4235-9948-40c4cd979c8c","Type":"ContainerDied","Data":"f1374736f51931bd80e29c1227d64e9197d1b74666dffbbd00cc2a082864911c"} Sep 11 00:01:23.440288 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:23.440227 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw" event={"ID":"f7a78418-f2e2-4235-9948-40c4cd979c8c","Type":"ContainerDied","Data":"c3dc59b40d9f0147d3e1ef4ae44348f262f477d0be423d0da46d3d7943f786dd"} Sep 11 00:01:23.440288 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:23.440250 2583 scope.go:117] "RemoveContainer" containerID="f1374736f51931bd80e29c1227d64e9197d1b74666dffbbd00cc2a082864911c" Sep 11 00:01:23.449139 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:23.449112 2583 scope.go:117] "RemoveContainer" containerID="6aa1c7bbd33a6f7c9a71d3eb9300bd191b72ad12b729857fc062088eaf2c56b5" Sep 11 00:01:23.457650 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:23.457626 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw"] Sep 11 00:01:23.457835 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:23.457765 2583 scope.go:117] "RemoveContainer" containerID="f1374736f51931bd80e29c1227d64e9197d1b74666dffbbd00cc2a082864911c" Sep 11 00:01:23.458040 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:01:23.458019 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1374736f51931bd80e29c1227d64e9197d1b74666dffbbd00cc2a082864911c\": container with ID starting with f1374736f51931bd80e29c1227d64e9197d1b74666dffbbd00cc2a082864911c not found: ID does not exist" containerID="f1374736f51931bd80e29c1227d64e9197d1b74666dffbbd00cc2a082864911c" Sep 11 00:01:23.458132 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:23.458052 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1374736f51931bd80e29c1227d64e9197d1b74666dffbbd00cc2a082864911c"} err="failed to get container status \"f1374736f51931bd80e29c1227d64e9197d1b74666dffbbd00cc2a082864911c\": rpc error: code = NotFound desc = could not find container \"f1374736f51931bd80e29c1227d64e9197d1b74666dffbbd00cc2a082864911c\": container with ID starting with f1374736f51931bd80e29c1227d64e9197d1b74666dffbbd00cc2a082864911c not found: ID does not exist" Sep 11 00:01:23.458132 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:23.458100 2583 scope.go:117] "RemoveContainer" containerID="6aa1c7bbd33a6f7c9a71d3eb9300bd191b72ad12b729857fc062088eaf2c56b5" Sep 11 00:01:23.458443 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:01:23.458422 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa1c7bbd33a6f7c9a71d3eb9300bd191b72ad12b729857fc062088eaf2c56b5\": container with ID starting with 6aa1c7bbd33a6f7c9a71d3eb9300bd191b72ad12b729857fc062088eaf2c56b5 not found: ID does not exist" containerID="6aa1c7bbd33a6f7c9a71d3eb9300bd191b72ad12b729857fc062088eaf2c56b5" Sep 11 00:01:23.458539 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:23.458446 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa1c7bbd33a6f7c9a71d3eb9300bd191b72ad12b729857fc062088eaf2c56b5"} err="failed to get container status \"6aa1c7bbd33a6f7c9a71d3eb9300bd191b72ad12b729857fc062088eaf2c56b5\": rpc error: code = NotFound desc = could not find container \"6aa1c7bbd33a6f7c9a71d3eb9300bd191b72ad12b729857fc062088eaf2c56b5\": container with ID starting with 6aa1c7bbd33a6f7c9a71d3eb9300bd191b72ad12b729857fc062088eaf2c56b5 not found: ID does not exist" Sep 11 00:01:23.463017 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:23.462997 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-02bac9-openshift-default-7c8d694c56-c5bzw"] Sep 11 00:01:25.307164 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:25.307132 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" path="/var/lib/kubelet/pods/f7a78418-f2e2-4235-9948-40c4cd979c8c/volumes" Sep 11 00:01:28.394654 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:28.394620 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj"] Sep 11 00:01:28.396420 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:28.396393 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396319-29818080-bqcsj"] Sep 11 00:01:28.427432 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:28.427402 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559"] Sep 11 00:01:28.427677 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:28.427625 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" podUID="c941c0e9-d399-4756-8dbc-4246de2604d0" containerName="payload-pre-processing" containerID="cri-o://fa3b3587212dd36bcef7eef867dbec58ae944cfeb7a6e2bda2acb92dfafb079a" gracePeriod=30 Sep 11 00:01:28.581151 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:28.581119 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn"] Sep 11 00:01:28.584686 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:28.584662 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396319-wf2rn"] Sep 11 00:01:28.685937 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:28.685637 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" Sep 11 00:01:28.822399 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:28.822360 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c941c0e9-d399-4756-8dbc-4246de2604d0-plugins-config-volume\") pod \"c941c0e9-d399-4756-8dbc-4246de2604d0\" (UID: \"c941c0e9-d399-4756-8dbc-4246de2604d0\") " Sep 11 00:01:28.822566 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:28.822417 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4kbd\" (UniqueName: \"kubernetes.io/projected/c941c0e9-d399-4756-8dbc-4246de2604d0-kube-api-access-l4kbd\") pod \"c941c0e9-d399-4756-8dbc-4246de2604d0\" (UID: \"c941c0e9-d399-4756-8dbc-4246de2604d0\") " Sep 11 00:01:28.822756 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:28.822728 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c941c0e9-d399-4756-8dbc-4246de2604d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c941c0e9-d399-4756-8dbc-4246de2604d0" (UID: "c941c0e9-d399-4756-8dbc-4246de2604d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:01:28.824425 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:28.824401 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c941c0e9-d399-4756-8dbc-4246de2604d0-kube-api-access-l4kbd" (OuterVolumeSpecName: "kube-api-access-l4kbd") pod "c941c0e9-d399-4756-8dbc-4246de2604d0" (UID: "c941c0e9-d399-4756-8dbc-4246de2604d0"). InnerVolumeSpecName "kube-api-access-l4kbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:01:28.923402 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:28.923316 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c941c0e9-d399-4756-8dbc-4246de2604d0-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:28.923402 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:28.923347 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4kbd\" (UniqueName: \"kubernetes.io/projected/c941c0e9-d399-4756-8dbc-4246de2604d0-kube-api-access-l4kbd\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:29.307543 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:29.307461 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="602d8cdf-5ff9-4d49-96b5-af35bc71eb08" path="/var/lib/kubelet/pods/602d8cdf-5ff9-4d49-96b5-af35bc71eb08/volumes" Sep 11 00:01:29.307762 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:29.307749 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1d858ad-b5ef-4390-a81b-2459f1dd367f" path="/var/lib/kubelet/pods/e1d858ad-b5ef-4390-a81b-2459f1dd367f/volumes" Sep 11 00:01:29.477327 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:29.477289 2583 generic.go:356] "Generic (PLEG): container finished" podID="c941c0e9-d399-4756-8dbc-4246de2604d0" containerID="fa3b3587212dd36bcef7eef867dbec58ae944cfeb7a6e2bda2acb92dfafb079a" exitCode=0 Sep 11 00:01:29.477713 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:29.477334 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" Sep 11 00:01:29.477713 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:29.477374 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" event={"ID":"c941c0e9-d399-4756-8dbc-4246de2604d0","Type":"ContainerDied","Data":"fa3b3587212dd36bcef7eef867dbec58ae944cfeb7a6e2bda2acb92dfafb079a"} Sep 11 00:01:29.477713 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:29.477429 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559" event={"ID":"c941c0e9-d399-4756-8dbc-4246de2604d0","Type":"ContainerDied","Data":"1f7c89c0003c94e9473d227d73b13187ad0daa5bac96924041bd18ccf0921006"} Sep 11 00:01:29.477713 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:29.477457 2583 scope.go:117] "RemoveContainer" containerID="fa3b3587212dd36bcef7eef867dbec58ae944cfeb7a6e2bda2acb92dfafb079a" Sep 11 00:01:29.486100 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:29.486080 2583 scope.go:117] "RemoveContainer" containerID="fa3b3587212dd36bcef7eef867dbec58ae944cfeb7a6e2bda2acb92dfafb079a" Sep 11 00:01:29.486353 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:01:29.486333 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3b3587212dd36bcef7eef867dbec58ae944cfeb7a6e2bda2acb92dfafb079a\": container with ID starting with fa3b3587212dd36bcef7eef867dbec58ae944cfeb7a6e2bda2acb92dfafb079a not found: ID does not exist" containerID="fa3b3587212dd36bcef7eef867dbec58ae944cfeb7a6e2bda2acb92dfafb079a" Sep 11 00:01:29.486401 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:29.486361 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3b3587212dd36bcef7eef867dbec58ae944cfeb7a6e2bda2acb92dfafb079a"} err="failed to get container status \"fa3b3587212dd36bcef7eef867dbec58ae944cfeb7a6e2bda2acb92dfafb079a\": rpc error: code = NotFound desc = could not find container \"fa3b3587212dd36bcef7eef867dbec58ae944cfeb7a6e2bda2acb92dfafb079a\": container with ID starting with fa3b3587212dd36bcef7eef867dbec58ae944cfeb7a6e2bda2acb92dfafb079a not found: ID does not exist" Sep 11 00:01:29.492904 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:29.492881 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559"] Sep 11 00:01:29.498208 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:29.498186 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396319-56bdc65b74-s5559"] Sep 11 00:01:31.307944 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:31.307909 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c941c0e9-d399-4756-8dbc-4246de2604d0" path="/var/lib/kubelet/pods/c941c0e9-d399-4756-8dbc-4246de2604d0/volumes" Sep 11 00:01:33.384194 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:33.384162 2583 scope.go:117] "RemoveContainer" containerID="29d72ccca5b1187d41321851cf5913e44fac906dc7aa0124d5a999bbd3f590c5" Sep 11 00:01:33.392174 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:33.392151 2583 scope.go:117] "RemoveContainer" containerID="612672f25e6737fc78545c1d5a22adf4074fcacae29a1ecc1e54c274d6665f33" Sep 11 00:01:34.825199 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:34.825165 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs"] Sep 11 00:01:34.825677 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:34.825432 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" containerID="cri-o://bc78727d178bb690b943c998b89c7f0f5fd18f8d53ef9523c27067669357d237" gracePeriod=30 Sep 11 00:01:35.179560 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.179526 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t"] Sep 11 00:01:35.179885 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.179873 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerName="istio-proxy" Sep 11 00:01:35.179933 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.179887 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerName="istio-proxy" Sep 11 00:01:35.179933 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.179899 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="602d8cdf-5ff9-4d49-96b5-af35bc71eb08" containerName="revoke-keys" Sep 11 00:01:35.179933 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.179905 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="602d8cdf-5ff9-4d49-96b5-af35bc71eb08" containerName="revoke-keys" Sep 11 00:01:35.179933 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.179916 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c941c0e9-d399-4756-8dbc-4246de2604d0" containerName="payload-pre-processing" Sep 11 00:01:35.179933 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.179922 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c941c0e9-d399-4756-8dbc-4246de2604d0" containerName="payload-pre-processing" Sep 11 00:01:35.180114 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.179982 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="602d8cdf-5ff9-4d49-96b5-af35bc71eb08" containerName="revoke-keys" Sep 11 00:01:35.180114 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.179991 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c941c0e9-d399-4756-8dbc-4246de2604d0" containerName="payload-pre-processing" Sep 11 00:01:35.180114 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.179996 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerName="istio-proxy" Sep 11 00:01:35.180114 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.180003 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerName="istio-proxy" Sep 11 00:01:35.182952 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.182937 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t" Sep 11 00:01:35.185514 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.185487 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:01:35.185634 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.185487 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 11 00:01:35.190856 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.190834 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t"] Sep 11 00:01:35.276887 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.276852 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8106bdd2-3557-4b76-a1ed-a5ca23eb5a70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t\" (UID: \"8106bdd2-3557-4b76-a1ed-a5ca23eb5a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t" Sep 11 00:01:35.378238 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.378206 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8106bdd2-3557-4b76-a1ed-a5ca23eb5a70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t\" (UID: \"8106bdd2-3557-4b76-a1ed-a5ca23eb5a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t" Sep 11 00:01:35.378702 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.378682 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8106bdd2-3557-4b76-a1ed-a5ca23eb5a70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t\" (UID: \"8106bdd2-3557-4b76-a1ed-a5ca23eb5a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t" Sep 11 00:01:35.494222 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.494135 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t" Sep 11 00:01:35.766857 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:35.766834 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t"] Sep 11 00:01:36.515159 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:36.515129 2583 generic.go:356] "Generic (PLEG): container finished" podID="8106bdd2-3557-4b76-a1ed-a5ca23eb5a70" containerID="93ff89558c95d1e00ae18e3295537e8531986d1832d238ae7f200074e8eef1de" exitCode=0 Sep 11 00:01:36.515497 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:36.515217 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t" event={"ID":"8106bdd2-3557-4b76-a1ed-a5ca23eb5a70","Type":"ContainerDied","Data":"93ff89558c95d1e00ae18e3295537e8531986d1832d238ae7f200074e8eef1de"} Sep 11 00:01:36.515497 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:36.515264 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t" event={"ID":"8106bdd2-3557-4b76-a1ed-a5ca23eb5a70","Type":"ContainerStarted","Data":"35436d6f8ad15bce8e984978a0dfc17cb06ba4145309312ccaea87cf7878045b"} Sep 11 00:01:37.589355 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:37.589316 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 00:01:37.589698 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:37.589389 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 00:01:37.648349 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:37.648326 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t" Sep 11 00:01:37.697666 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:37.697635 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8106bdd2-3557-4b76-a1ed-a5ca23eb5a70-maas-api-service-ca\") pod \"8106bdd2-3557-4b76-a1ed-a5ca23eb5a70\" (UID: \"8106bdd2-3557-4b76-a1ed-a5ca23eb5a70\") " Sep 11 00:01:37.697988 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:37.697954 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8106bdd2-3557-4b76-a1ed-a5ca23eb5a70-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8106bdd2-3557-4b76-a1ed-a5ca23eb5a70" (UID: "8106bdd2-3557-4b76-a1ed-a5ca23eb5a70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:01:37.798719 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:37.798627 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8106bdd2-3557-4b76-a1ed-a5ca23eb5a70-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:38.526967 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:38.526937 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t" Sep 11 00:01:38.526967 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:38.526950 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t" event={"ID":"8106bdd2-3557-4b76-a1ed-a5ca23eb5a70","Type":"ContainerDied","Data":"35436d6f8ad15bce8e984978a0dfc17cb06ba4145309312ccaea87cf7878045b"} Sep 11 00:01:38.527228 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:38.527006 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35436d6f8ad15bce8e984978a0dfc17cb06ba4145309312ccaea87cf7878045b" Sep 11 00:01:40.069258 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.069236 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 11 00:01:40.119102 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119055 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-workload-certs\") pod \"71c463d7-e804-4097-a2e3-c6a67d900db4\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " Sep 11 00:01:40.119283 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119112 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-token\") pod \"71c463d7-e804-4097-a2e3-c6a67d900db4\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " Sep 11 00:01:40.119283 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119144 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c463d7-e804-4097-a2e3-c6a67d900db4-istiod-ca-cert\") pod \"71c463d7-e804-4097-a2e3-c6a67d900db4\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " Sep 11 00:01:40.119283 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119180 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-credential-socket\") pod \"71c463d7-e804-4097-a2e3-c6a67d900db4\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " Sep 11 00:01:40.119283 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119200 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-data\") pod \"71c463d7-e804-4097-a2e3-c6a67d900db4\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " Sep 11 00:01:40.119283 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119246 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-podinfo\") pod \"71c463d7-e804-4097-a2e3-c6a67d900db4\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " Sep 11 00:01:40.119283 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119273 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-envoy\") pod \"71c463d7-e804-4097-a2e3-c6a67d900db4\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " Sep 11 00:01:40.119623 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119316 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-workload-socket\") pod \"71c463d7-e804-4097-a2e3-c6a67d900db4\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " Sep 11 00:01:40.119623 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119325 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71c463d7-e804-4097-a2e3-c6a67d900db4" (UID: "71c463d7-e804-4097-a2e3-c6a67d900db4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:40.119623 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119346 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmrn7\" (UniqueName: \"kubernetes.io/projected/71c463d7-e804-4097-a2e3-c6a67d900db4-kube-api-access-pmrn7\") pod \"71c463d7-e804-4097-a2e3-c6a67d900db4\" (UID: \"71c463d7-e804-4097-a2e3-c6a67d900db4\") " Sep 11 00:01:40.119623 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119439 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71c463d7-e804-4097-a2e3-c6a67d900db4" (UID: "71c463d7-e804-4097-a2e3-c6a67d900db4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:40.119623 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119520 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-data" (OuterVolumeSpecName: "istio-data") pod "71c463d7-e804-4097-a2e3-c6a67d900db4" (UID: "71c463d7-e804-4097-a2e3-c6a67d900db4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:40.119623 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119562 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c463d7-e804-4097-a2e3-c6a67d900db4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71c463d7-e804-4097-a2e3-c6a67d900db4" (UID: "71c463d7-e804-4097-a2e3-c6a67d900db4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:01:40.119924 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119694 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:40.119924 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119756 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:40.119924 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119769 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:40.119924 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119781 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c463d7-e804-4097-a2e3-c6a67d900db4-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:40.119924 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.119816 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71c463d7-e804-4097-a2e3-c6a67d900db4" (UID: "71c463d7-e804-4097-a2e3-c6a67d900db4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:40.121478 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.121447 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71c463d7-e804-4097-a2e3-c6a67d900db4" (UID: "71c463d7-e804-4097-a2e3-c6a67d900db4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:40.121605 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.121549 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71c463d7-e804-4097-a2e3-c6a67d900db4" (UID: "71c463d7-e804-4097-a2e3-c6a67d900db4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:01:40.121605 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.121564 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c463d7-e804-4097-a2e3-c6a67d900db4-kube-api-access-pmrn7" (OuterVolumeSpecName: "kube-api-access-pmrn7") pod "71c463d7-e804-4097-a2e3-c6a67d900db4" (UID: "71c463d7-e804-4097-a2e3-c6a67d900db4"). InnerVolumeSpecName "kube-api-access-pmrn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:01:40.121605 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.121585 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-token" (OuterVolumeSpecName: "istio-token") pod "71c463d7-e804-4097-a2e3-c6a67d900db4" (UID: "71c463d7-e804-4097-a2e3-c6a67d900db4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:01:40.220299 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.220262 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:40.220299 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.220296 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:40.220523 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.220309 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c463d7-e804-4097-a2e3-c6a67d900db4-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:40.220523 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.220323 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmrn7\" (UniqueName: \"kubernetes.io/projected/71c463d7-e804-4097-a2e3-c6a67d900db4-kube-api-access-pmrn7\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:40.220523 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.220338 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c463d7-e804-4097-a2e3-c6a67d900db4-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:40.540788 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.540660 2583 generic.go:356] "Generic (PLEG): container finished" podID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerID="bc78727d178bb690b943c998b89c7f0f5fd18f8d53ef9523c27067669357d237" exitCode=0 Sep 11 00:01:40.541018 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.540720 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" event={"ID":"71c463d7-e804-4097-a2e3-c6a67d900db4","Type":"ContainerDied","Data":"bc78727d178bb690b943c998b89c7f0f5fd18f8d53ef9523c27067669357d237"} Sep 11 00:01:40.541261 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.540756 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" Sep 11 00:01:40.541388 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.541239 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs" event={"ID":"71c463d7-e804-4097-a2e3-c6a67d900db4","Type":"ContainerDied","Data":"28240618e9d718c466041ce0fdb905d70e5637ce41de250ce4407c03e586a3d8"} Sep 11 00:01:40.541451 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.541259 2583 scope.go:117] "RemoveContainer" containerID="bc78727d178bb690b943c998b89c7f0f5fd18f8d53ef9523c27067669357d237" Sep 11 00:01:40.552794 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.552775 2583 scope.go:117] "RemoveContainer" containerID="321a76ad05daec47fa528cfe1097c221125366d48abfd4326b495cca059a4304" Sep 11 00:01:40.560424 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.560408 2583 scope.go:117] "RemoveContainer" containerID="bc78727d178bb690b943c998b89c7f0f5fd18f8d53ef9523c27067669357d237" Sep 11 00:01:40.560696 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:01:40.560677 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc78727d178bb690b943c998b89c7f0f5fd18f8d53ef9523c27067669357d237\": container with ID starting with bc78727d178bb690b943c998b89c7f0f5fd18f8d53ef9523c27067669357d237 not found: ID does not exist" containerID="bc78727d178bb690b943c998b89c7f0f5fd18f8d53ef9523c27067669357d237" Sep 11 00:01:40.560743 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.560705 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc78727d178bb690b943c998b89c7f0f5fd18f8d53ef9523c27067669357d237"} err="failed to get container status \"bc78727d178bb690b943c998b89c7f0f5fd18f8d53ef9523c27067669357d237\": rpc error: code = NotFound desc = could not find container \"bc78727d178bb690b943c998b89c7f0f5fd18f8d53ef9523c27067669357d237\": container with ID starting with bc78727d178bb690b943c998b89c7f0f5fd18f8d53ef9523c27067669357d237 not found: ID does not exist" Sep 11 00:01:40.560743 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.560723 2583 scope.go:117] "RemoveContainer" containerID="321a76ad05daec47fa528cfe1097c221125366d48abfd4326b495cca059a4304" Sep 11 00:01:40.560978 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:01:40.560964 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321a76ad05daec47fa528cfe1097c221125366d48abfd4326b495cca059a4304\": container with ID starting with 321a76ad05daec47fa528cfe1097c221125366d48abfd4326b495cca059a4304 not found: ID does not exist" containerID="321a76ad05daec47fa528cfe1097c221125366d48abfd4326b495cca059a4304" Sep 11 00:01:40.561022 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.560982 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321a76ad05daec47fa528cfe1097c221125366d48abfd4326b495cca059a4304"} err="failed to get container status \"321a76ad05daec47fa528cfe1097c221125366d48abfd4326b495cca059a4304\": rpc error: code = NotFound desc = could not find container \"321a76ad05daec47fa528cfe1097c221125366d48abfd4326b495cca059a4304\": container with ID starting with 321a76ad05daec47fa528cfe1097c221125366d48abfd4326b495cca059a4304 not found: ID does not exist" Sep 11 00:01:40.564360 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.564336 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs"] Sep 11 00:01:40.569886 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:40.569865 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396319-openshift-default-5ccfb47976-475qs"] Sep 11 00:01:41.308022 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:41.307978 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" path="/var/lib/kubelet/pods/71c463d7-e804-4097-a2e3-c6a67d900db4/volumes" Sep 11 00:01:46.022268 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.022211 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5"] Sep 11 00:01:46.026201 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.026176 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-16e45a-29818080-wlgk5"] Sep 11 00:01:46.044642 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.044612 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2"] Sep 11 00:01:46.044887 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.044861 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" podUID="7cc0c12c-3aa3-4b95-895c-088d21f03987" containerName="payload-processing" containerID="cri-o://9186cf9549183c1be196287d12da136b003bfc45f2f2bbdd6bba400ae23c92ac" gracePeriod=30 Sep 11 00:01:46.054239 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.054213 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6"] Sep 11 00:01:46.054516 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.054452 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" podUID="7b61aec6-a470-4f6a-a4b0-a7c388feebac" containerName="payload-pre-processing" containerID="cri-o://ff43b7bb881097cb9d794a63b0067668ac46260dc7395cf1c0dbedc49abd63e2" gracePeriod=30 Sep 11 00:01:46.208764 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.208732 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t"] Sep 11 00:01:46.213657 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.213619 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-16e45a-65n9t"] Sep 11 00:01:46.332635 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.332611 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" Sep 11 00:01:46.335188 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.335168 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" Sep 11 00:01:46.371844 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.371818 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvmpm\" (UniqueName: \"kubernetes.io/projected/7cc0c12c-3aa3-4b95-895c-088d21f03987-kube-api-access-lvmpm\") pod \"7cc0c12c-3aa3-4b95-895c-088d21f03987\" (UID: \"7cc0c12c-3aa3-4b95-895c-088d21f03987\") " Sep 11 00:01:46.371948 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.371859 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b61aec6-a470-4f6a-a4b0-a7c388feebac-plugins-config-volume\") pod \"7b61aec6-a470-4f6a-a4b0-a7c388feebac\" (UID: \"7b61aec6-a470-4f6a-a4b0-a7c388feebac\") " Sep 11 00:01:46.372024 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.372000 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cc0c12c-3aa3-4b95-895c-088d21f03987-plugins-config-volume\") pod \"7cc0c12c-3aa3-4b95-895c-088d21f03987\" (UID: \"7cc0c12c-3aa3-4b95-895c-088d21f03987\") " Sep 11 00:01:46.372102 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.372088 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-667nq\" (UniqueName: \"kubernetes.io/projected/7b61aec6-a470-4f6a-a4b0-a7c388feebac-kube-api-access-667nq\") pod \"7b61aec6-a470-4f6a-a4b0-a7c388feebac\" (UID: \"7b61aec6-a470-4f6a-a4b0-a7c388feebac\") " Sep 11 00:01:46.372360 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.372333 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b61aec6-a470-4f6a-a4b0-a7c388feebac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b61aec6-a470-4f6a-a4b0-a7c388feebac" (UID: "7b61aec6-a470-4f6a-a4b0-a7c388feebac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:01:46.372500 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.372367 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc0c12c-3aa3-4b95-895c-088d21f03987-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7cc0c12c-3aa3-4b95-895c-088d21f03987" (UID: "7cc0c12c-3aa3-4b95-895c-088d21f03987"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:01:46.374380 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.374354 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b61aec6-a470-4f6a-a4b0-a7c388feebac-kube-api-access-667nq" (OuterVolumeSpecName: "kube-api-access-667nq") pod "7b61aec6-a470-4f6a-a4b0-a7c388feebac" (UID: "7b61aec6-a470-4f6a-a4b0-a7c388feebac"). InnerVolumeSpecName "kube-api-access-667nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:01:46.374651 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.374625 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc0c12c-3aa3-4b95-895c-088d21f03987-kube-api-access-lvmpm" (OuterVolumeSpecName: "kube-api-access-lvmpm") pod "7cc0c12c-3aa3-4b95-895c-088d21f03987" (UID: "7cc0c12c-3aa3-4b95-895c-088d21f03987"). InnerVolumeSpecName "kube-api-access-lvmpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:01:46.472883 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.472847 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-667nq\" (UniqueName: \"kubernetes.io/projected/7b61aec6-a470-4f6a-a4b0-a7c388feebac-kube-api-access-667nq\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:46.472883 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.472877 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvmpm\" (UniqueName: \"kubernetes.io/projected/7cc0c12c-3aa3-4b95-895c-088d21f03987-kube-api-access-lvmpm\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:46.472883 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.472887 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b61aec6-a470-4f6a-a4b0-a7c388feebac-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:46.473135 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.472895 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cc0c12c-3aa3-4b95-895c-088d21f03987-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:01:46.573133 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.573043 2583 generic.go:356] "Generic (PLEG): container finished" podID="7cc0c12c-3aa3-4b95-895c-088d21f03987" containerID="9186cf9549183c1be196287d12da136b003bfc45f2f2bbdd6bba400ae23c92ac" exitCode=0 Sep 11 00:01:46.573133 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.573110 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" event={"ID":"7cc0c12c-3aa3-4b95-895c-088d21f03987","Type":"ContainerDied","Data":"9186cf9549183c1be196287d12da136b003bfc45f2f2bbdd6bba400ae23c92ac"} Sep 11 00:01:46.573338 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.573176 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" event={"ID":"7cc0c12c-3aa3-4b95-895c-088d21f03987","Type":"ContainerDied","Data":"4ce1790c71ed16303db1bcd92981a5faaa5a93a7ecee45592ff22a85c002186b"} Sep 11 00:01:46.573338 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.573128 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2" Sep 11 00:01:46.573338 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.573206 2583 scope.go:117] "RemoveContainer" containerID="9186cf9549183c1be196287d12da136b003bfc45f2f2bbdd6bba400ae23c92ac" Sep 11 00:01:46.574652 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.574626 2583 generic.go:356] "Generic (PLEG): container finished" podID="7b61aec6-a470-4f6a-a4b0-a7c388feebac" containerID="ff43b7bb881097cb9d794a63b0067668ac46260dc7395cf1c0dbedc49abd63e2" exitCode=0 Sep 11 00:01:46.574788 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.574665 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" Sep 11 00:01:46.574788 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.574671 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" event={"ID":"7b61aec6-a470-4f6a-a4b0-a7c388feebac","Type":"ContainerDied","Data":"ff43b7bb881097cb9d794a63b0067668ac46260dc7395cf1c0dbedc49abd63e2"} Sep 11 00:01:46.574788 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.574714 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6" event={"ID":"7b61aec6-a470-4f6a-a4b0-a7c388feebac","Type":"ContainerDied","Data":"f47ad3b88bdad557f54ee9a8bb834cc661cf572d685fcfd1797ec150eb5f16c6"} Sep 11 00:01:46.582674 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.582557 2583 scope.go:117] "RemoveContainer" containerID="9186cf9549183c1be196287d12da136b003bfc45f2f2bbdd6bba400ae23c92ac" Sep 11 00:01:46.582941 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:01:46.582917 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9186cf9549183c1be196287d12da136b003bfc45f2f2bbdd6bba400ae23c92ac\": container with ID starting with 9186cf9549183c1be196287d12da136b003bfc45f2f2bbdd6bba400ae23c92ac not found: ID does not exist" containerID="9186cf9549183c1be196287d12da136b003bfc45f2f2bbdd6bba400ae23c92ac" Sep 11 00:01:46.583054 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.582946 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9186cf9549183c1be196287d12da136b003bfc45f2f2bbdd6bba400ae23c92ac"} err="failed to get container status \"9186cf9549183c1be196287d12da136b003bfc45f2f2bbdd6bba400ae23c92ac\": rpc error: code = NotFound desc = could not find container \"9186cf9549183c1be196287d12da136b003bfc45f2f2bbdd6bba400ae23c92ac\": container with ID starting with 9186cf9549183c1be196287d12da136b003bfc45f2f2bbdd6bba400ae23c92ac not found: ID does not exist" Sep 11 00:01:46.583054 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.582970 2583 scope.go:117] "RemoveContainer" containerID="ff43b7bb881097cb9d794a63b0067668ac46260dc7395cf1c0dbedc49abd63e2" Sep 11 00:01:46.590772 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.590756 2583 scope.go:117] "RemoveContainer" containerID="ff43b7bb881097cb9d794a63b0067668ac46260dc7395cf1c0dbedc49abd63e2" Sep 11 00:01:46.591040 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:01:46.591019 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff43b7bb881097cb9d794a63b0067668ac46260dc7395cf1c0dbedc49abd63e2\": container with ID starting with ff43b7bb881097cb9d794a63b0067668ac46260dc7395cf1c0dbedc49abd63e2 not found: ID does not exist" containerID="ff43b7bb881097cb9d794a63b0067668ac46260dc7395cf1c0dbedc49abd63e2" Sep 11 00:01:46.591120 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.591050 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff43b7bb881097cb9d794a63b0067668ac46260dc7395cf1c0dbedc49abd63e2"} err="failed to get container status \"ff43b7bb881097cb9d794a63b0067668ac46260dc7395cf1c0dbedc49abd63e2\": rpc error: code = NotFound desc = could not find container \"ff43b7bb881097cb9d794a63b0067668ac46260dc7395cf1c0dbedc49abd63e2\": container with ID starting with ff43b7bb881097cb9d794a63b0067668ac46260dc7395cf1c0dbedc49abd63e2 not found: ID does not exist" Sep 11 00:01:46.597555 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.597533 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6"] Sep 11 00:01:46.604180 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.604157 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-16e45a-797675546-c6pp6"] Sep 11 00:01:46.613325 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.613306 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2"] Sep 11 00:01:46.616913 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:46.616894 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-16e45a-76bcd8f699-jwjj2"] Sep 11 00:01:47.308124 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:47.308058 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b61aec6-a470-4f6a-a4b0-a7c388feebac" path="/var/lib/kubelet/pods/7b61aec6-a470-4f6a-a4b0-a7c388feebac/volumes" Sep 11 00:01:47.308632 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:47.308611 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cc0c12c-3aa3-4b95-895c-088d21f03987" path="/var/lib/kubelet/pods/7cc0c12c-3aa3-4b95-895c-088d21f03987/volumes" Sep 11 00:01:47.308979 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:47.308961 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8106bdd2-3557-4b76-a1ed-a5ca23eb5a70" path="/var/lib/kubelet/pods/8106bdd2-3557-4b76-a1ed-a5ca23eb5a70/volumes" Sep 11 00:01:47.309280 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:01:47.309263 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="accca1f6-83a9-4bdf-8f50-67ad74569188" path="/var/lib/kubelet/pods/accca1f6-83a9-4bdf-8f50-67ad74569188/volumes" Sep 11 00:02:33.424708 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:02:33.424676 2583 scope.go:117] "RemoveContainer" containerID="1d349e59bb60caa29235427689c311f9b6fbb3c02d3bdc1948423110e1222d9a" Sep 11 00:02:33.432419 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:02:33.432397 2583 scope.go:117] "RemoveContainer" containerID="c19c0dbae5e3be9c4e32eaa1d663258dedbad48a310a940472a218dfe5f64c33" Sep 11 00:03:15.667598 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.667564 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58"] Sep 11 00:03:15.668136 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668025 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" Sep 11 00:03:15.668136 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668039 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" Sep 11 00:03:15.668136 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668055 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" Sep 11 00:03:15.668136 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668064 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" Sep 11 00:03:15.668136 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668103 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc0c12c-3aa3-4b95-895c-088d21f03987" containerName="payload-processing" Sep 11 00:03:15.668136 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668112 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc0c12c-3aa3-4b95-895c-088d21f03987" containerName="payload-processing" Sep 11 00:03:15.668136 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668126 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerName="istio-proxy" Sep 11 00:03:15.668136 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668134 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7a78418-f2e2-4235-9948-40c4cd979c8c" containerName="istio-proxy" Sep 11 00:03:15.668405 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668145 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b61aec6-a470-4f6a-a4b0-a7c388feebac" containerName="payload-pre-processing" Sep 11 00:03:15.668405 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668153 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b61aec6-a470-4f6a-a4b0-a7c388feebac" containerName="payload-pre-processing" Sep 11 00:03:15.668405 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668193 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8106bdd2-3557-4b76-a1ed-a5ca23eb5a70" containerName="revoke-keys" Sep 11 00:03:15.668405 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668202 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8106bdd2-3557-4b76-a1ed-a5ca23eb5a70" containerName="revoke-keys" Sep 11 00:03:15.668405 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668281 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8106bdd2-3557-4b76-a1ed-a5ca23eb5a70" containerName="revoke-keys" Sep 11 00:03:15.668405 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668294 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc0c12c-3aa3-4b95-895c-088d21f03987" containerName="payload-processing" Sep 11 00:03:15.668405 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668303 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" Sep 11 00:03:15.668405 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668310 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b61aec6-a470-4f6a-a4b0-a7c388feebac" containerName="payload-pre-processing" Sep 11 00:03:15.668405 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.668319 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c463d7-e804-4097-a2e3-c6a67d900db4" containerName="istio-proxy" Sep 11 00:03:15.671425 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.671406 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.673725 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.673702 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7c3755-openshift-default-dockercfg-6skgc\"" Sep 11 00:03:15.682737 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.682710 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58"] Sep 11 00:03:15.809687 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.809650 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmjp6\" (UniqueName: \"kubernetes.io/projected/c1dbf00a-72c9-41c5-a00d-657096586bb4-kube-api-access-jmjp6\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.809687 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.809693 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dbf00a-72c9-41c5-a00d-657096586bb4-istiod-ca-cert\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.809960 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.809714 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-envoy\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.809960 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.809792 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-workload-socket\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.809960 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.809853 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-podinfo\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.809960 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.809888 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-data\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.809960 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.809906 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-credential-socket\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.810255 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.809984 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-workload-certs\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.810255 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.810008 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-token\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.911172 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.911140 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-workload-socket\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.911374 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.911188 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-podinfo\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.911374 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.911288 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-data\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.911374 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.911308 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-credential-socket\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.911374 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.911333 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-workload-certs\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.911374 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.911351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-token\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.911659 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.911512 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmjp6\" (UniqueName: \"kubernetes.io/projected/c1dbf00a-72c9-41c5-a00d-657096586bb4-kube-api-access-jmjp6\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.911659 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.911563 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dbf00a-72c9-41c5-a00d-657096586bb4-istiod-ca-cert\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.911659 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.911598 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-envoy\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.911659 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.911602 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-workload-socket\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.911873 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.911658 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-credential-socket\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.911873 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.911732 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-data\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.911984 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.911881 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-workload-certs\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.912258 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.912239 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dbf00a-72c9-41c5-a00d-657096586bb4-istiod-ca-cert\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.913790 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.913773 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-envoy\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.913790 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.913782 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-podinfo\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.918875 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.918804 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-token\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.919034 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.919016 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmjp6\" (UniqueName: \"kubernetes.io/projected/c1dbf00a-72c9-41c5-a00d-657096586bb4-kube-api-access-jmjp6\") pod \"e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:15.984887 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:15.984857 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:16.257804 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:16.257760 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58"] Sep 11 00:03:16.260120 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:03:16.260089 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dbf00a_72c9_41c5_a00d_657096586bb4.slice/crio-4499cb62635ce6c016913d66d514ddc60c9495eec05de9703dbd32f2f22a61b2 WatchSource:0}: Error finding container 4499cb62635ce6c016913d66d514ddc60c9495eec05de9703dbd32f2f22a61b2: Status 404 returned error can't find the container with id 4499cb62635ce6c016913d66d514ddc60c9495eec05de9703dbd32f2f22a61b2 Sep 11 00:03:16.262263 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:16.262229 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:03:16.262384 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:16.262306 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:03:16.262384 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:16.262350 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:03:17.021278 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:17.021239 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" event={"ID":"c1dbf00a-72c9-41c5-a00d-657096586bb4","Type":"ContainerStarted","Data":"e520c1cd09e1fd735bd156ec1a698b188bf86798f7a2c213cef105c31e346ab1"} Sep 11 00:03:17.021643 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:17.021303 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" event={"ID":"c1dbf00a-72c9-41c5-a00d-657096586bb4","Type":"ContainerStarted","Data":"4499cb62635ce6c016913d66d514ddc60c9495eec05de9703dbd32f2f22a61b2"} Sep 11 00:03:17.042886 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:17.042833 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" podStartSLOduration=2.0428104 podStartE2EDuration="2.0428104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:17.04166527 +0000 UTC m=+1604.344143273" watchObservedRunningTime="2026-09-11 00:03:17.0428104 +0000 UTC m=+1604.345288412" Sep 11 00:03:17.985470 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:17.985438 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:17.990866 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:17.990841 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:18.026247 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:18.026219 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:18.027506 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:18.027479 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:21.473335 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:21.473305 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5"] Sep 11 00:03:21.476428 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:21.476412 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" Sep 11 00:03:21.478610 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:21.478591 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7c3755\"" Sep 11 00:03:21.488044 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:21.488024 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5"] Sep 11 00:03:21.665248 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:21.665208 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a2f27e-f5cc-4350-a9b1-5ade4071e0ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5\" (UID: \"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" Sep 11 00:03:21.665423 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:21.665254 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt8sn\" (UniqueName: \"kubernetes.io/projected/72a2f27e-f5cc-4350-a9b1-5ade4071e0ec-kube-api-access-tt8sn\") pod \"payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5\" (UID: \"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" Sep 11 00:03:21.766551 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:21.766470 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a2f27e-f5cc-4350-a9b1-5ade4071e0ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5\" (UID: \"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" Sep 11 00:03:21.766551 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:21.766513 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt8sn\" (UniqueName: \"kubernetes.io/projected/72a2f27e-f5cc-4350-a9b1-5ade4071e0ec-kube-api-access-tt8sn\") pod \"payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5\" (UID: \"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" Sep 11 00:03:21.767238 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:21.767217 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a2f27e-f5cc-4350-a9b1-5ade4071e0ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5\" (UID: \"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" Sep 11 00:03:21.774487 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:21.774466 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt8sn\" (UniqueName: \"kubernetes.io/projected/72a2f27e-f5cc-4350-a9b1-5ade4071e0ec-kube-api-access-tt8sn\") pod \"payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5\" (UID: \"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" Sep 11 00:03:21.786366 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:21.786344 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" Sep 11 00:03:22.055101 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:22.054992 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5"] Sep 11 00:03:22.057741 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:03:22.057714 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a2f27e_f5cc_4350_a9b1_5ade4071e0ec.slice/crio-9a8955c6fb5f9d95d1af5617664bf7720b35a2cd3c5e9abec5aa09380ca7cf08 WatchSource:0}: Error finding container 9a8955c6fb5f9d95d1af5617664bf7720b35a2cd3c5e9abec5aa09380ca7cf08: Status 404 returned error can't find the container with id 9a8955c6fb5f9d95d1af5617664bf7720b35a2cd3c5e9abec5aa09380ca7cf08 Sep 11 00:03:23.053220 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:23.053181 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" event={"ID":"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec","Type":"ContainerStarted","Data":"5aed6ca021bf62dbb8b782743cd6d5c0e14f482747bb0721e41637c482348c19"} Sep 11 00:03:23.053706 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:23.053236 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" event={"ID":"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec","Type":"ContainerStarted","Data":"9a8955c6fb5f9d95d1af5617664bf7720b35a2cd3c5e9abec5aa09380ca7cf08"} Sep 11 00:03:23.053706 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:23.053268 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" Sep 11 00:03:23.071523 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:23.071479 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" podStartSLOduration=2.071462543 podStartE2EDuration="2.071462543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:23.069200195 +0000 UTC m=+1610.371678211" watchObservedRunningTime="2026-09-11 00:03:23.071462543 +0000 UTC m=+1610.373940554" Sep 11 00:03:33.463879 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:33.463846 2583 scope.go:117] "RemoveContainer" containerID="62c0700334384f5c636f8d2d5012357bfafa85bc49586f7f0b7737c9774ae287" Sep 11 00:03:34.060742 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:34.060707 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" Sep 11 00:03:34.068191 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:34.068166 2583 patch_prober.go:28] interesting pod/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 00:03:34.068277 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:34.068210 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 00:03:35.120553 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:35.120518 2583 generic.go:356] "Generic (PLEG): container finished" podID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerID="e520c1cd09e1fd735bd156ec1a698b188bf86798f7a2c213cef105c31e346ab1" exitCode=0 Sep 11 00:03:35.121096 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:35.120596 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" event={"ID":"c1dbf00a-72c9-41c5-a00d-657096586bb4","Type":"ContainerDied","Data":"e520c1cd09e1fd735bd156ec1a698b188bf86798f7a2c213cef105c31e346ab1"} Sep 11 00:03:35.121183 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:35.121145 2583 scope.go:117] "RemoveContainer" containerID="e520c1cd09e1fd735bd156ec1a698b188bf86798f7a2c213cef105c31e346ab1" Sep 11 00:03:35.122222 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:35.122197 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:03:35.122294 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:35.122259 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:03:35.122332 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:35.122293 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:03:36.131047 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:36.131000 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" event={"ID":"c1dbf00a-72c9-41c5-a00d-657096586bb4","Type":"ContainerStarted","Data":"acd4f87379a1439b03b4ece29e42615e2a2fc056ffb49c277ec02a0d317f179e"} Sep 11 00:03:36.986093 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:36.986041 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:36.986297 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:36.986278 2583 patch_prober.go:28] interesting pod/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 11 00:03:36.986349 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:36.986321 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 11 00:03:37.985582 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:37.985539 2583 patch_prober.go:28] interesting pod/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 11 00:03:37.985993 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:37.985600 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 11 00:03:38.985237 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:38.985201 2583 patch_prober.go:28] interesting pod/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 11 00:03:38.985415 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:38.985257 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 11 00:03:39.985896 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:39.985855 2583 patch_prober.go:28] interesting pod/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 11 00:03:39.986317 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:39.985924 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 11 00:03:40.986018 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:40.985973 2583 patch_prober.go:28] interesting pod/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 11 00:03:40.986455 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:40.986039 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 11 00:03:41.988694 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:41.988661 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:41.989099 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:41.989035 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:41.989794 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:41.989775 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:03:51.860743 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:51.860701 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp"] Sep 11 00:03:51.864413 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:51.864390 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" Sep 11 00:03:51.867351 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:51.867332 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-da6814-dockercfg-dlq6x\"" Sep 11 00:03:51.867707 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:51.867689 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-da6814\"" Sep 11 00:03:51.875364 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:51.875343 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp"] Sep 11 00:03:51.937090 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:51.937039 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6pfh\" (UniqueName: \"kubernetes.io/projected/2a347ab7-4db8-4712-9ddb-787b730efe86-kube-api-access-p6pfh\") pod \"payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp\" (UID: \"2a347ab7-4db8-4712-9ddb-787b730efe86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" Sep 11 00:03:51.937233 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:51.937168 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a347ab7-4db8-4712-9ddb-787b730efe86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp\" (UID: \"2a347ab7-4db8-4712-9ddb-787b730efe86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" Sep 11 00:03:52.038105 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:52.038045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a347ab7-4db8-4712-9ddb-787b730efe86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp\" (UID: \"2a347ab7-4db8-4712-9ddb-787b730efe86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" Sep 11 00:03:52.038263 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:52.038137 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6pfh\" (UniqueName: \"kubernetes.io/projected/2a347ab7-4db8-4712-9ddb-787b730efe86-kube-api-access-p6pfh\") pod \"payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp\" (UID: \"2a347ab7-4db8-4712-9ddb-787b730efe86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" Sep 11 00:03:52.038742 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:52.038721 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a347ab7-4db8-4712-9ddb-787b730efe86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp\" (UID: \"2a347ab7-4db8-4712-9ddb-787b730efe86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" Sep 11 00:03:52.056709 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:52.056680 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6pfh\" (UniqueName: \"kubernetes.io/projected/2a347ab7-4db8-4712-9ddb-787b730efe86-kube-api-access-p6pfh\") pod \"payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp\" (UID: \"2a347ab7-4db8-4712-9ddb-787b730efe86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" Sep 11 00:03:52.175855 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:52.175758 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" Sep 11 00:03:52.444823 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:52.444794 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp"] Sep 11 00:03:52.447217 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:03:52.447183 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a347ab7_4db8_4712_9ddb_787b730efe86.slice/crio-c4ce8ca346e402cbaa31d94aa62635507c37b0b7cf177af5ed44005d79d2bf66 WatchSource:0}: Error finding container c4ce8ca346e402cbaa31d94aa62635507c37b0b7cf177af5ed44005d79d2bf66: Status 404 returned error can't find the container with id c4ce8ca346e402cbaa31d94aa62635507c37b0b7cf177af5ed44005d79d2bf66 Sep 11 00:03:53.226236 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:53.226196 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" event={"ID":"2a347ab7-4db8-4712-9ddb-787b730efe86","Type":"ContainerStarted","Data":"d961cd4971bd9b30f84b642f6145fda88eb7560d5563fc5dd279cd2446b44dcd"} Sep 11 00:03:53.226621 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:53.226267 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" Sep 11 00:03:53.226621 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:53.226290 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" event={"ID":"2a347ab7-4db8-4712-9ddb-787b730efe86","Type":"ContainerStarted","Data":"c4ce8ca346e402cbaa31d94aa62635507c37b0b7cf177af5ed44005d79d2bf66"} Sep 11 00:03:53.243047 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:03:53.242999 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" podStartSLOduration=2.242983085 podStartE2EDuration="2.242983085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:53.241333518 +0000 UTC m=+1640.543811532" watchObservedRunningTime="2026-09-11 00:03:53.242983085 +0000 UTC m=+1640.545461129" Sep 11 00:04:04.233492 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:04:04.233456 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" Sep 11 00:04:59.965114 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:04:59.965007 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7"] Sep 11 00:04:59.968548 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:04:59.968529 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:04:59.971064 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:04:59.971041 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-ef4657-openshift-default-dockercfg-k7969\"" Sep 11 00:04:59.980147 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:04:59.980125 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7"] Sep 11 00:05:00.126560 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.126526 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-workload-certs\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.126560 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.126567 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-data\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.126809 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.126591 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.126809 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.126667 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-token\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.126809 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.126706 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-envoy\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.126809 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.126746 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-workload-socket\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.126809 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.126765 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-credential-socket\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.126809 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.126783 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4dwm\" (UniqueName: \"kubernetes.io/projected/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-kube-api-access-r4dwm\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.127025 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.126813 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-podinfo\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.228090 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.227975 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4dwm\" (UniqueName: \"kubernetes.io/projected/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-kube-api-access-r4dwm\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.228090 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.228018 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-podinfo\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.228090 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.228064 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-workload-certs\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.228410 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.228106 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-data\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.228410 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.228130 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.228410 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.228165 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-token\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.228410 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.228206 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-envoy\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.228410 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.228249 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-workload-socket\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.228410 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.228270 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-credential-socket\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.228752 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.228537 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-data\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.228895 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.228870 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-credential-socket\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.228970 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.228939 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-workload-certs\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.229025 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.228966 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-workload-socket\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.229275 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.229250 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.230574 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.230546 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-envoy\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.230574 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.230565 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-podinfo\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.235926 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.235903 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-token\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.236026 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.235973 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4dwm\" (UniqueName: \"kubernetes.io/projected/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-kube-api-access-r4dwm\") pod \"e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.280232 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.280201 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:00.556764 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.556740 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7"] Sep 11 00:05:00.558400 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:05:00.558370 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2f4598_4e0d_4c30_a810_b49146aa7ffd.slice/crio-f81035ab494247f980009c66660624609f72560262c59854e87642ee8bfbb7c1 WatchSource:0}: Error finding container f81035ab494247f980009c66660624609f72560262c59854e87642ee8bfbb7c1: Status 404 returned error can't find the container with id f81035ab494247f980009c66660624609f72560262c59854e87642ee8bfbb7c1 Sep 11 00:05:00.560444 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.560408 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:05:00.560570 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.560474 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:05:00.560570 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:00.560502 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:05:01.553706 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:01.553666 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" event={"ID":"fa2f4598-4e0d-4c30-a810-b49146aa7ffd","Type":"ContainerStarted","Data":"03dfddeaec899df27b8d34d74c78e4c2f733b07b9c104f58614fd35d1946178d"} Sep 11 00:05:01.554121 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:01.553731 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" event={"ID":"fa2f4598-4e0d-4c30-a810-b49146aa7ffd","Type":"ContainerStarted","Data":"f81035ab494247f980009c66660624609f72560262c59854e87642ee8bfbb7c1"} Sep 11 00:05:01.573900 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:01.573850 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" podStartSLOduration=2.573833588 podStartE2EDuration="2.573833588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:05:01.572048909 +0000 UTC m=+1708.874526917" watchObservedRunningTime="2026-09-11 00:05:01.573833588 +0000 UTC m=+1708.876311599" Sep 11 00:05:02.280989 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:02.280955 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:02.286016 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:02.285990 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:02.559215 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:02.559131 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:02.560087 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:02.560056 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:07.564767 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:07.564729 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp"] Sep 11 00:05:07.568230 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:07.568206 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" Sep 11 00:05:07.570748 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:07.570726 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-ef4657-dockercfg-rvn54\"" Sep 11 00:05:07.571022 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:07.571007 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ef4657\"" Sep 11 00:05:07.580462 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:07.580442 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp"] Sep 11 00:05:07.697875 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:07.697823 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2c3d44-7765-4c73-9c99-aeea65e26b67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp\" (UID: \"bf2c3d44-7765-4c73-9c99-aeea65e26b67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" Sep 11 00:05:07.698290 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:07.698261 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxfwb\" (UniqueName: \"kubernetes.io/projected/bf2c3d44-7765-4c73-9c99-aeea65e26b67-kube-api-access-sxfwb\") pod \"payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp\" (UID: \"bf2c3d44-7765-4c73-9c99-aeea65e26b67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" Sep 11 00:05:07.799747 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:07.799711 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxfwb\" (UniqueName: \"kubernetes.io/projected/bf2c3d44-7765-4c73-9c99-aeea65e26b67-kube-api-access-sxfwb\") pod \"payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp\" (UID: \"bf2c3d44-7765-4c73-9c99-aeea65e26b67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" Sep 11 00:05:07.799927 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:07.799774 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2c3d44-7765-4c73-9c99-aeea65e26b67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp\" (UID: \"bf2c3d44-7765-4c73-9c99-aeea65e26b67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" Sep 11 00:05:07.800464 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:07.800444 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2c3d44-7765-4c73-9c99-aeea65e26b67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp\" (UID: \"bf2c3d44-7765-4c73-9c99-aeea65e26b67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" Sep 11 00:05:07.808128 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:07.808107 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxfwb\" (UniqueName: \"kubernetes.io/projected/bf2c3d44-7765-4c73-9c99-aeea65e26b67-kube-api-access-sxfwb\") pod \"payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp\" (UID: \"bf2c3d44-7765-4c73-9c99-aeea65e26b67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" Sep 11 00:05:07.879124 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:07.879065 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" Sep 11 00:05:08.148932 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:08.148903 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp"] Sep 11 00:05:08.151517 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:05:08.151485 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2c3d44_7765_4c73_9c99_aeea65e26b67.slice/crio-6fea23ab6fac7817d646c9b3915789d65db5555545de33d1fad8314201b6b741 WatchSource:0}: Error finding container 6fea23ab6fac7817d646c9b3915789d65db5555545de33d1fad8314201b6b741: Status 404 returned error can't find the container with id 6fea23ab6fac7817d646c9b3915789d65db5555545de33d1fad8314201b6b741 Sep 11 00:05:08.588457 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:08.588372 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" event={"ID":"bf2c3d44-7765-4c73-9c99-aeea65e26b67","Type":"ContainerStarted","Data":"524099b5abf478ae16a8a7f7dbcf11bbf0634d3e4d3cf65f8c29ed7915f8e798"} Sep 11 00:05:08.588457 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:08.588426 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" event={"ID":"bf2c3d44-7765-4c73-9c99-aeea65e26b67","Type":"ContainerStarted","Data":"6fea23ab6fac7817d646c9b3915789d65db5555545de33d1fad8314201b6b741"} Sep 11 00:05:08.588831 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:08.588515 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" Sep 11 00:05:08.604423 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:08.604379 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" podStartSLOduration=1.6043649279999999 podStartE2EDuration="1.604364928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:05:08.603240642 +0000 UTC m=+1715.905718654" watchObservedRunningTime="2026-09-11 00:05:08.604364928 +0000 UTC m=+1715.906842967" Sep 11 00:05:19.595906 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:19.595820 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" Sep 11 00:05:22.671163 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:22.671126 2583 generic.go:356] "Generic (PLEG): container finished" podID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerID="03dfddeaec899df27b8d34d74c78e4c2f733b07b9c104f58614fd35d1946178d" exitCode=0 Sep 11 00:05:22.671578 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:22.671200 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" event={"ID":"fa2f4598-4e0d-4c30-a810-b49146aa7ffd","Type":"ContainerDied","Data":"03dfddeaec899df27b8d34d74c78e4c2f733b07b9c104f58614fd35d1946178d"} Sep 11 00:05:22.671653 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:22.671638 2583 scope.go:117] "RemoveContainer" containerID="03dfddeaec899df27b8d34d74c78e4c2f733b07b9c104f58614fd35d1946178d" Sep 11 00:05:22.672749 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:22.672718 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:05:22.673131 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:22.672806 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:05:22.673131 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:22.672852 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:05:23.677709 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:23.677663 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" event={"ID":"fa2f4598-4e0d-4c30-a810-b49146aa7ffd","Type":"ContainerStarted","Data":"3aafda73aa9060c9348bfa7ee5cca18e62370c651887f813ce76dce70b764737"} Sep 11 00:05:24.280502 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:24.280463 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:24.280719 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:24.280695 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 00:05:24.280767 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:24.280746 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" podUID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 00:05:25.281544 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:25.281513 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 00:05:25.281947 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:25.281565 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" podUID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 00:05:26.281477 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:26.281442 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 00:05:26.281657 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:26.281494 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" podUID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 00:05:27.281258 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:27.281221 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 00:05:27.281425 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:27.281278 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" podUID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 00:05:28.281576 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:28.281538 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 00:05:28.281954 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:28.281612 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" podUID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 00:05:29.316888 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:29.316858 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:29.317384 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:29.317226 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:29.318106 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:29.318084 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:34.206640 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:34.206606 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh"] Sep 11 00:05:34.210319 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:34.210301 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh" Sep 11 00:05:34.212837 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:34.212816 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:05:34.212961 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:34.212943 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 11 00:05:34.216770 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:34.216746 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh"] Sep 11 00:05:34.341783 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:34.341752 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73168f22-62a2-47fd-925e-3072e65dd505-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh\" (UID: \"73168f22-62a2-47fd-925e-3072e65dd505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh" Sep 11 00:05:34.442375 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:34.442340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73168f22-62a2-47fd-925e-3072e65dd505-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh\" (UID: \"73168f22-62a2-47fd-925e-3072e65dd505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh" Sep 11 00:05:34.442970 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:34.442950 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73168f22-62a2-47fd-925e-3072e65dd505-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh\" (UID: \"73168f22-62a2-47fd-925e-3072e65dd505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh" Sep 11 00:05:34.522204 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:34.522129 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh" Sep 11 00:05:34.792972 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:34.792949 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh"] Sep 11 00:05:34.794836 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:05:34.794788 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73168f22_62a2_47fd_925e_3072e65dd505.slice/crio-8bb52924c4b6b6d038c075459bc2862f2cc9d5dc7891a587d71994fa237aa101 WatchSource:0}: Error finding container 8bb52924c4b6b6d038c075459bc2862f2cc9d5dc7891a587d71994fa237aa101: Status 404 returned error can't find the container with id 8bb52924c4b6b6d038c075459bc2862f2cc9d5dc7891a587d71994fa237aa101 Sep 11 00:05:35.736906 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:35.736868 2583 generic.go:356] "Generic (PLEG): container finished" podID="73168f22-62a2-47fd-925e-3072e65dd505" containerID="100d887aabfce9fe08a09dbbae99d571bda8d3167ba65087b5dc33d19454d2c9" exitCode=0 Sep 11 00:05:35.737304 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:35.736919 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh" event={"ID":"73168f22-62a2-47fd-925e-3072e65dd505","Type":"ContainerDied","Data":"100d887aabfce9fe08a09dbbae99d571bda8d3167ba65087b5dc33d19454d2c9"} Sep 11 00:05:35.737304 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:35.736973 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh" event={"ID":"73168f22-62a2-47fd-925e-3072e65dd505","Type":"ContainerStarted","Data":"8bb52924c4b6b6d038c075459bc2862f2cc9d5dc7891a587d71994fa237aa101"} Sep 11 00:05:36.911133 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:36.911109 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh" Sep 11 00:05:36.965598 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:36.965571 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73168f22-62a2-47fd-925e-3072e65dd505-maas-api-service-ca\") pod \"73168f22-62a2-47fd-925e-3072e65dd505\" (UID: \"73168f22-62a2-47fd-925e-3072e65dd505\") " Sep 11 00:05:36.965931 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:36.965904 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73168f22-62a2-47fd-925e-3072e65dd505-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73168f22-62a2-47fd-925e-3072e65dd505" (UID: "73168f22-62a2-47fd-925e-3072e65dd505"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:05:37.066487 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:37.066407 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73168f22-62a2-47fd-925e-3072e65dd505-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:05:37.753358 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:37.753311 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh" event={"ID":"73168f22-62a2-47fd-925e-3072e65dd505","Type":"ContainerDied","Data":"8bb52924c4b6b6d038c075459bc2862f2cc9d5dc7891a587d71994fa237aa101"} Sep 11 00:05:37.753541 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:37.753374 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh" Sep 11 00:05:37.753541 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:37.753421 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bb52924c4b6b6d038c075459bc2862f2cc9d5dc7891a587d71994fa237aa101" Sep 11 00:05:45.251934 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.251895 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp"] Sep 11 00:05:45.252537 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.252174 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" podUID="bf2c3d44-7765-4c73-9c99-aeea65e26b67" containerName="payload-pre-processing" containerID="cri-o://524099b5abf478ae16a8a7f7dbcf11bbf0634d3e4d3cf65f8c29ed7915f8e798" gracePeriod=30 Sep 11 00:05:45.411419 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.411384 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh"] Sep 11 00:05:45.415209 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.415172 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ef4657-58zbh"] Sep 11 00:05:45.533372 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.533347 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" Sep 11 00:05:45.649317 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.649277 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxfwb\" (UniqueName: \"kubernetes.io/projected/bf2c3d44-7765-4c73-9c99-aeea65e26b67-kube-api-access-sxfwb\") pod \"bf2c3d44-7765-4c73-9c99-aeea65e26b67\" (UID: \"bf2c3d44-7765-4c73-9c99-aeea65e26b67\") " Sep 11 00:05:45.649515 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.649364 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2c3d44-7765-4c73-9c99-aeea65e26b67-plugins-config-volume\") pod \"bf2c3d44-7765-4c73-9c99-aeea65e26b67\" (UID: \"bf2c3d44-7765-4c73-9c99-aeea65e26b67\") " Sep 11 00:05:45.649737 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.649703 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2c3d44-7765-4c73-9c99-aeea65e26b67-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf2c3d44-7765-4c73-9c99-aeea65e26b67" (UID: "bf2c3d44-7765-4c73-9c99-aeea65e26b67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:05:45.651398 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.651359 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2c3d44-7765-4c73-9c99-aeea65e26b67-kube-api-access-sxfwb" (OuterVolumeSpecName: "kube-api-access-sxfwb") pod "bf2c3d44-7765-4c73-9c99-aeea65e26b67" (UID: "bf2c3d44-7765-4c73-9c99-aeea65e26b67"). InnerVolumeSpecName "kube-api-access-sxfwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:05:45.750104 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.750050 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxfwb\" (UniqueName: \"kubernetes.io/projected/bf2c3d44-7765-4c73-9c99-aeea65e26b67-kube-api-access-sxfwb\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:05:45.750104 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.750090 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2c3d44-7765-4c73-9c99-aeea65e26b67-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:05:45.796465 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.796369 2583 generic.go:356] "Generic (PLEG): container finished" podID="bf2c3d44-7765-4c73-9c99-aeea65e26b67" containerID="524099b5abf478ae16a8a7f7dbcf11bbf0634d3e4d3cf65f8c29ed7915f8e798" exitCode=0 Sep 11 00:05:45.796465 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.796415 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" Sep 11 00:05:45.796465 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.796449 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" event={"ID":"bf2c3d44-7765-4c73-9c99-aeea65e26b67","Type":"ContainerDied","Data":"524099b5abf478ae16a8a7f7dbcf11bbf0634d3e4d3cf65f8c29ed7915f8e798"} Sep 11 00:05:45.796743 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.796489 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp" event={"ID":"bf2c3d44-7765-4c73-9c99-aeea65e26b67","Type":"ContainerDied","Data":"6fea23ab6fac7817d646c9b3915789d65db5555545de33d1fad8314201b6b741"} Sep 11 00:05:45.796743 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.796515 2583 scope.go:117] "RemoveContainer" containerID="524099b5abf478ae16a8a7f7dbcf11bbf0634d3e4d3cf65f8c29ed7915f8e798" Sep 11 00:05:45.805582 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.805565 2583 scope.go:117] "RemoveContainer" containerID="524099b5abf478ae16a8a7f7dbcf11bbf0634d3e4d3cf65f8c29ed7915f8e798" Sep 11 00:05:45.805866 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:05:45.805847 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524099b5abf478ae16a8a7f7dbcf11bbf0634d3e4d3cf65f8c29ed7915f8e798\": container with ID starting with 524099b5abf478ae16a8a7f7dbcf11bbf0634d3e4d3cf65f8c29ed7915f8e798 not found: ID does not exist" containerID="524099b5abf478ae16a8a7f7dbcf11bbf0634d3e4d3cf65f8c29ed7915f8e798" Sep 11 00:05:45.805910 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.805877 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524099b5abf478ae16a8a7f7dbcf11bbf0634d3e4d3cf65f8c29ed7915f8e798"} err="failed to get container status \"524099b5abf478ae16a8a7f7dbcf11bbf0634d3e4d3cf65f8c29ed7915f8e798\": rpc error: code = NotFound desc = could not find container \"524099b5abf478ae16a8a7f7dbcf11bbf0634d3e4d3cf65f8c29ed7915f8e798\": container with ID starting with 524099b5abf478ae16a8a7f7dbcf11bbf0634d3e4d3cf65f8c29ed7915f8e798 not found: ID does not exist" Sep 11 00:05:45.817740 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.817716 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp"] Sep 11 00:05:45.821294 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:45.821274 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ef4657-5489dc75c8-tdtlp"] Sep 11 00:05:47.307850 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:47.307814 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73168f22-62a2-47fd-925e-3072e65dd505" path="/var/lib/kubelet/pods/73168f22-62a2-47fd-925e-3072e65dd505/volumes" Sep 11 00:05:47.308252 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:47.308202 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf2c3d44-7765-4c73-9c99-aeea65e26b67" path="/var/lib/kubelet/pods/bf2c3d44-7765-4c73-9c99-aeea65e26b67/volumes" Sep 11 00:05:52.329372 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:52.329337 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7"] Sep 11 00:05:52.329752 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:52.329594 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" podUID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerName="istio-proxy" containerID="cri-o://3aafda73aa9060c9348bfa7ee5cca18e62370c651887f813ce76dce70b764737" gracePeriod=30 Sep 11 00:05:55.789007 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.788970 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7"] Sep 11 00:05:55.789403 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.789343 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73168f22-62a2-47fd-925e-3072e65dd505" containerName="revoke-keys" Sep 11 00:05:55.789403 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.789357 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="73168f22-62a2-47fd-925e-3072e65dd505" containerName="revoke-keys" Sep 11 00:05:55.789403 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.789372 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf2c3d44-7765-4c73-9c99-aeea65e26b67" containerName="payload-pre-processing" Sep 11 00:05:55.789403 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.789378 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf2c3d44-7765-4c73-9c99-aeea65e26b67" containerName="payload-pre-processing" Sep 11 00:05:55.789571 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.789441 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf2c3d44-7765-4c73-9c99-aeea65e26b67" containerName="payload-pre-processing" Sep 11 00:05:55.789571 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.789451 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="73168f22-62a2-47fd-925e-3072e65dd505" containerName="revoke-keys" Sep 11 00:05:55.793410 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.793394 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7" Sep 11 00:05:55.795860 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.795829 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 11 00:05:55.796009 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.795867 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:05:55.800297 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.800236 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7"] Sep 11 00:05:55.840508 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.840473 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d33506-b28e-4f56-9d9e-4097922c0677-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7\" (UID: \"f4d33506-b28e-4f56-9d9e-4097922c0677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7" Sep 11 00:05:55.941285 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.941254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d33506-b28e-4f56-9d9e-4097922c0677-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7\" (UID: \"f4d33506-b28e-4f56-9d9e-4097922c0677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7" Sep 11 00:05:55.941898 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:55.941877 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d33506-b28e-4f56-9d9e-4097922c0677-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7\" (UID: \"f4d33506-b28e-4f56-9d9e-4097922c0677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7" Sep 11 00:05:56.104344 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:56.104318 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7" Sep 11 00:05:56.373632 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:56.373607 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7"] Sep 11 00:05:56.375795 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:05:56.375770 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d33506_b28e_4f56_9d9e_4097922c0677.slice/crio-e3b858a7874dbf89d79f2c0eac19e1202b0fd35eda19b27fdecf73b94ad0673b WatchSource:0}: Error finding container e3b858a7874dbf89d79f2c0eac19e1202b0fd35eda19b27fdecf73b94ad0673b: Status 404 returned error can't find the container with id e3b858a7874dbf89d79f2c0eac19e1202b0fd35eda19b27fdecf73b94ad0673b Sep 11 00:05:56.853513 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:56.853471 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7" event={"ID":"f4d33506-b28e-4f56-9d9e-4097922c0677","Type":"ContainerStarted","Data":"2592c2d645733ca7abe0a4e9357d90a08c2bd50654d9e9eebb64bde05ba01219"} Sep 11 00:05:56.853947 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:56.853530 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7" event={"ID":"f4d33506-b28e-4f56-9d9e-4097922c0677","Type":"ContainerStarted","Data":"e3b858a7874dbf89d79f2c0eac19e1202b0fd35eda19b27fdecf73b94ad0673b"} Sep 11 00:05:56.868486 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:56.868438 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7" podStartSLOduration=1.868423635 podStartE2EDuration="1.868423635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:05:56.867371432 +0000 UTC m=+1764.169849445" watchObservedRunningTime="2026-09-11 00:05:56.868423635 +0000 UTC m=+1764.170901648" Sep 11 00:05:57.573044 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.573017 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:57.655183 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655150 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istiod-ca-cert\") pod \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " Sep 11 00:05:57.655380 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655191 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-workload-socket\") pod \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " Sep 11 00:05:57.655380 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655225 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4dwm\" (UniqueName: \"kubernetes.io/projected/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-kube-api-access-r4dwm\") pod \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " Sep 11 00:05:57.655380 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655253 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-envoy\") pod \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " Sep 11 00:05:57.655380 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655308 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-credential-socket\") pod \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " Sep 11 00:05:57.655380 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655334 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-data\") pod \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " Sep 11 00:05:57.655660 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655392 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-podinfo\") pod \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " Sep 11 00:05:57.655660 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655456 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-token\") pod \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " Sep 11 00:05:57.655660 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655467 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fa2f4598-4e0d-4c30-a810-b49146aa7ffd" (UID: "fa2f4598-4e0d-4c30-a810-b49146aa7ffd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:57.655660 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655604 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fa2f4598-4e0d-4c30-a810-b49146aa7ffd" (UID: "fa2f4598-4e0d-4c30-a810-b49146aa7ffd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:57.655660 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655646 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-data" (OuterVolumeSpecName: "istio-data") pod "fa2f4598-4e0d-4c30-a810-b49146aa7ffd" (UID: "fa2f4598-4e0d-4c30-a810-b49146aa7ffd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:57.655660 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655658 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fa2f4598-4e0d-4c30-a810-b49146aa7ffd" (UID: "fa2f4598-4e0d-4c30-a810-b49146aa7ffd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:05:57.655964 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655483 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-workload-certs\") pod \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\" (UID: \"fa2f4598-4e0d-4c30-a810-b49146aa7ffd\") " Sep 11 00:05:57.655964 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.655845 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fa2f4598-4e0d-4c30-a810-b49146aa7ffd" (UID: "fa2f4598-4e0d-4c30-a810-b49146aa7ffd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:57.656175 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.656146 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:05:57.656175 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.656169 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:05:57.656302 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.656185 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:05:57.656302 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.656198 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:05:57.656302 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.656211 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:05:57.657575 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.657546 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fa2f4598-4e0d-4c30-a810-b49146aa7ffd" (UID: "fa2f4598-4e0d-4c30-a810-b49146aa7ffd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:57.657670 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.657573 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fa2f4598-4e0d-4c30-a810-b49146aa7ffd" (UID: "fa2f4598-4e0d-4c30-a810-b49146aa7ffd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:05:57.657879 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.657852 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-kube-api-access-r4dwm" (OuterVolumeSpecName: "kube-api-access-r4dwm") pod "fa2f4598-4e0d-4c30-a810-b49146aa7ffd" (UID: "fa2f4598-4e0d-4c30-a810-b49146aa7ffd"). InnerVolumeSpecName "kube-api-access-r4dwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:05:57.657879 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.657865 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-token" (OuterVolumeSpecName: "istio-token") pod "fa2f4598-4e0d-4c30-a810-b49146aa7ffd" (UID: "fa2f4598-4e0d-4c30-a810-b49146aa7ffd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:05:57.757027 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.756998 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:05:57.757027 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.757024 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:05:57.757027 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.757033 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4dwm\" (UniqueName: \"kubernetes.io/projected/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-kube-api-access-r4dwm\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:05:57.757244 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.757042 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2f4598-4e0d-4c30-a810-b49146aa7ffd-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:05:57.860277 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.860245 2583 generic.go:356] "Generic (PLEG): container finished" podID="f4d33506-b28e-4f56-9d9e-4097922c0677" containerID="2592c2d645733ca7abe0a4e9357d90a08c2bd50654d9e9eebb64bde05ba01219" exitCode=0 Sep 11 00:05:57.860694 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.860333 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7" event={"ID":"f4d33506-b28e-4f56-9d9e-4097922c0677","Type":"ContainerDied","Data":"2592c2d645733ca7abe0a4e9357d90a08c2bd50654d9e9eebb64bde05ba01219"} Sep 11 00:05:57.862019 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.861995 2583 generic.go:356] "Generic (PLEG): container finished" podID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerID="3aafda73aa9060c9348bfa7ee5cca18e62370c651887f813ce76dce70b764737" exitCode=0 Sep 11 00:05:57.862158 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.862037 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" Sep 11 00:05:57.862158 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.862088 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" event={"ID":"fa2f4598-4e0d-4c30-a810-b49146aa7ffd","Type":"ContainerDied","Data":"3aafda73aa9060c9348bfa7ee5cca18e62370c651887f813ce76dce70b764737"} Sep 11 00:05:57.862158 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.862133 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7" event={"ID":"fa2f4598-4e0d-4c30-a810-b49146aa7ffd","Type":"ContainerDied","Data":"f81035ab494247f980009c66660624609f72560262c59854e87642ee8bfbb7c1"} Sep 11 00:05:57.862158 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.862153 2583 scope.go:117] "RemoveContainer" containerID="3aafda73aa9060c9348bfa7ee5cca18e62370c651887f813ce76dce70b764737" Sep 11 00:05:57.878217 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.878194 2583 scope.go:117] "RemoveContainer" containerID="03dfddeaec899df27b8d34d74c78e4c2f733b07b9c104f58614fd35d1946178d" Sep 11 00:05:57.887027 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.886963 2583 scope.go:117] "RemoveContainer" containerID="3aafda73aa9060c9348bfa7ee5cca18e62370c651887f813ce76dce70b764737" Sep 11 00:05:57.887572 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:05:57.887543 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aafda73aa9060c9348bfa7ee5cca18e62370c651887f813ce76dce70b764737\": container with ID starting with 3aafda73aa9060c9348bfa7ee5cca18e62370c651887f813ce76dce70b764737 not found: ID does not exist" containerID="3aafda73aa9060c9348bfa7ee5cca18e62370c651887f813ce76dce70b764737" Sep 11 00:05:57.887665 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.887583 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aafda73aa9060c9348bfa7ee5cca18e62370c651887f813ce76dce70b764737"} err="failed to get container status \"3aafda73aa9060c9348bfa7ee5cca18e62370c651887f813ce76dce70b764737\": rpc error: code = NotFound desc = could not find container \"3aafda73aa9060c9348bfa7ee5cca18e62370c651887f813ce76dce70b764737\": container with ID starting with 3aafda73aa9060c9348bfa7ee5cca18e62370c651887f813ce76dce70b764737 not found: ID does not exist" Sep 11 00:05:57.887665 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.887606 2583 scope.go:117] "RemoveContainer" containerID="03dfddeaec899df27b8d34d74c78e4c2f733b07b9c104f58614fd35d1946178d" Sep 11 00:05:57.887856 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:05:57.887836 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03dfddeaec899df27b8d34d74c78e4c2f733b07b9c104f58614fd35d1946178d\": container with ID starting with 03dfddeaec899df27b8d34d74c78e4c2f733b07b9c104f58614fd35d1946178d not found: ID does not exist" containerID="03dfddeaec899df27b8d34d74c78e4c2f733b07b9c104f58614fd35d1946178d" Sep 11 00:05:57.887916 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.887863 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03dfddeaec899df27b8d34d74c78e4c2f733b07b9c104f58614fd35d1946178d"} err="failed to get container status \"03dfddeaec899df27b8d34d74c78e4c2f733b07b9c104f58614fd35d1946178d\": rpc error: code = NotFound desc = could not find container \"03dfddeaec899df27b8d34d74c78e4c2f733b07b9c104f58614fd35d1946178d\": container with ID starting with 03dfddeaec899df27b8d34d74c78e4c2f733b07b9c104f58614fd35d1946178d not found: ID does not exist" Sep 11 00:05:57.888746 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.888721 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7"] Sep 11 00:05:57.892524 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:57.892502 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ef4657-openshift-default-67597b87c8-mrpt7"] Sep 11 00:05:59.004714 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:59.004689 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7" Sep 11 00:05:59.070954 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:59.070922 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d33506-b28e-4f56-9d9e-4097922c0677-maas-api-service-ca\") pod \"f4d33506-b28e-4f56-9d9e-4097922c0677\" (UID: \"f4d33506-b28e-4f56-9d9e-4097922c0677\") " Sep 11 00:05:59.071340 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:59.071315 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d33506-b28e-4f56-9d9e-4097922c0677-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4d33506-b28e-4f56-9d9e-4097922c0677" (UID: "f4d33506-b28e-4f56-9d9e-4097922c0677"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:05:59.172561 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:59.172490 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d33506-b28e-4f56-9d9e-4097922c0677-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:05:59.308399 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:59.308359 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" path="/var/lib/kubelet/pods/fa2f4598-4e0d-4c30-a810-b49146aa7ffd/volumes" Sep 11 00:05:59.876208 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:59.876178 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7" Sep 11 00:05:59.876397 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:59.876223 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7" event={"ID":"f4d33506-b28e-4f56-9d9e-4097922c0677","Type":"ContainerDied","Data":"e3b858a7874dbf89d79f2c0eac19e1202b0fd35eda19b27fdecf73b94ad0673b"} Sep 11 00:05:59.876397 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:05:59.876271 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3b858a7874dbf89d79f2c0eac19e1202b0fd35eda19b27fdecf73b94ad0673b" Sep 11 00:06:11.387198 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.387165 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5"] Sep 11 00:06:11.387775 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.387428 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" podUID="72a2f27e-f5cc-4350-a9b1-5ade4071e0ec" containerName="payload-pre-processing" containerID="cri-o://5aed6ca021bf62dbb8b782743cd6d5c0e14f482747bb0721e41637c482348c19" gracePeriod=30 Sep 11 00:06:11.553551 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.553490 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7"] Sep 11 00:06:11.556451 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.556265 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c3755-zvkz7"] Sep 11 00:06:11.644908 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.644836 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" Sep 11 00:06:11.678683 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.678655 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a2f27e-f5cc-4350-a9b1-5ade4071e0ec-plugins-config-volume\") pod \"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec\" (UID: \"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec\") " Sep 11 00:06:11.678859 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.678689 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt8sn\" (UniqueName: \"kubernetes.io/projected/72a2f27e-f5cc-4350-a9b1-5ade4071e0ec-kube-api-access-tt8sn\") pod \"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec\" (UID: \"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec\") " Sep 11 00:06:11.679016 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.678991 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a2f27e-f5cc-4350-a9b1-5ade4071e0ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72a2f27e-f5cc-4350-a9b1-5ade4071e0ec" (UID: "72a2f27e-f5cc-4350-a9b1-5ade4071e0ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:06:11.680833 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.680800 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a2f27e-f5cc-4350-a9b1-5ade4071e0ec-kube-api-access-tt8sn" (OuterVolumeSpecName: "kube-api-access-tt8sn") pod "72a2f27e-f5cc-4350-a9b1-5ade4071e0ec" (UID: "72a2f27e-f5cc-4350-a9b1-5ade4071e0ec"). InnerVolumeSpecName "kube-api-access-tt8sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:06:11.779982 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.779946 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a2f27e-f5cc-4350-a9b1-5ade4071e0ec-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:06:11.779982 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.779977 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt8sn\" (UniqueName: \"kubernetes.io/projected/72a2f27e-f5cc-4350-a9b1-5ade4071e0ec-kube-api-access-tt8sn\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:06:11.938327 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.938237 2583 generic.go:356] "Generic (PLEG): container finished" podID="72a2f27e-f5cc-4350-a9b1-5ade4071e0ec" containerID="5aed6ca021bf62dbb8b782743cd6d5c0e14f482747bb0721e41637c482348c19" exitCode=0 Sep 11 00:06:11.938327 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.938284 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" Sep 11 00:06:11.938545 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.938324 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" event={"ID":"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec","Type":"ContainerDied","Data":"5aed6ca021bf62dbb8b782743cd6d5c0e14f482747bb0721e41637c482348c19"} Sep 11 00:06:11.938545 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.938376 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5" event={"ID":"72a2f27e-f5cc-4350-a9b1-5ade4071e0ec","Type":"ContainerDied","Data":"9a8955c6fb5f9d95d1af5617664bf7720b35a2cd3c5e9abec5aa09380ca7cf08"} Sep 11 00:06:11.938545 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.938400 2583 scope.go:117] "RemoveContainer" containerID="5aed6ca021bf62dbb8b782743cd6d5c0e14f482747bb0721e41637c482348c19" Sep 11 00:06:11.947572 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.947554 2583 scope.go:117] "RemoveContainer" containerID="5aed6ca021bf62dbb8b782743cd6d5c0e14f482747bb0721e41637c482348c19" Sep 11 00:06:11.947806 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:06:11.947786 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aed6ca021bf62dbb8b782743cd6d5c0e14f482747bb0721e41637c482348c19\": container with ID starting with 5aed6ca021bf62dbb8b782743cd6d5c0e14f482747bb0721e41637c482348c19 not found: ID does not exist" containerID="5aed6ca021bf62dbb8b782743cd6d5c0e14f482747bb0721e41637c482348c19" Sep 11 00:06:11.947877 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.947818 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aed6ca021bf62dbb8b782743cd6d5c0e14f482747bb0721e41637c482348c19"} err="failed to get container status \"5aed6ca021bf62dbb8b782743cd6d5c0e14f482747bb0721e41637c482348c19\": rpc error: code = NotFound desc = could not find container \"5aed6ca021bf62dbb8b782743cd6d5c0e14f482747bb0721e41637c482348c19\": container with ID starting with 5aed6ca021bf62dbb8b782743cd6d5c0e14f482747bb0721e41637c482348c19 not found: ID does not exist" Sep 11 00:06:11.958368 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.958343 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5"] Sep 11 00:06:11.960381 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:11.960360 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c3755-5f98fdcbc8-t8wv5"] Sep 11 00:06:13.307590 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:13.307556 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72a2f27e-f5cc-4350-a9b1-5ade4071e0ec" path="/var/lib/kubelet/pods/72a2f27e-f5cc-4350-a9b1-5ade4071e0ec/volumes" Sep 11 00:06:13.307960 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:13.307876 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4d33506-b28e-4f56-9d9e-4097922c0677" path="/var/lib/kubelet/pods/f4d33506-b28e-4f56-9d9e-4097922c0677/volumes" Sep 11 00:06:33.503380 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:33.503345 2583 scope.go:117] "RemoveContainer" containerID="290725d5e5e2c3386b77456b50740a4237d98363eed9c356c967c397c3cc0c45" Sep 11 00:06:33.511868 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:33.511847 2583 scope.go:117] "RemoveContainer" containerID="b2af44b19a5b0861e693383670efd456be1eb36080ddb0f6ba16d874abdadbed" Sep 11 00:06:33.519501 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:06:33.519481 2583 scope.go:117] "RemoveContainer" containerID="9b01e557f346bd9bba459511c6254bed28a66b16d0e17a0b9f1f38fddac74ece" Sep 11 00:07:14.109478 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.109443 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58"] Sep 11 00:07:14.109983 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.109655 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerName="istio-proxy" containerID="cri-o://acd4f87379a1439b03b4ece29e42615e2a2fc056ffb49c277ec02a0d317f179e" gracePeriod=30 Sep 11 00:07:14.505757 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.505668 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf"] Sep 11 00:07:14.506121 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.506103 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72a2f27e-f5cc-4350-a9b1-5ade4071e0ec" containerName="payload-pre-processing" Sep 11 00:07:14.506227 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.506123 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="72a2f27e-f5cc-4350-a9b1-5ade4071e0ec" containerName="payload-pre-processing" Sep 11 00:07:14.506227 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.506147 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4d33506-b28e-4f56-9d9e-4097922c0677" containerName="revoke-keys" Sep 11 00:07:14.506227 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.506155 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4d33506-b28e-4f56-9d9e-4097922c0677" containerName="revoke-keys" Sep 11 00:07:14.506227 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.506183 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerName="istio-proxy" Sep 11 00:07:14.506227 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.506192 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerName="istio-proxy" Sep 11 00:07:14.506227 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.506202 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerName="istio-proxy" Sep 11 00:07:14.506227 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.506209 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerName="istio-proxy" Sep 11 00:07:14.506558 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.506290 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerName="istio-proxy" Sep 11 00:07:14.506558 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.506300 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4d33506-b28e-4f56-9d9e-4097922c0677" containerName="revoke-keys" Sep 11 00:07:14.506558 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.506309 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="72a2f27e-f5cc-4350-a9b1-5ade4071e0ec" containerName="payload-pre-processing" Sep 11 00:07:14.506558 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.506320 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa2f4598-4e0d-4c30-a810-b49146aa7ffd" containerName="istio-proxy" Sep 11 00:07:14.509317 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.509296 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf" Sep 11 00:07:14.511772 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.511744 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:07:14.511881 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.511754 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 11 00:07:14.516772 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.516749 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf"] Sep 11 00:07:14.611707 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.611669 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39938499-3689-4d6f-8ce9-c84b0190fa84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf\" (UID: \"39938499-3689-4d6f-8ce9-c84b0190fa84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf" Sep 11 00:07:14.713028 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.712993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39938499-3689-4d6f-8ce9-c84b0190fa84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf\" (UID: \"39938499-3689-4d6f-8ce9-c84b0190fa84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf" Sep 11 00:07:14.713608 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.713590 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39938499-3689-4d6f-8ce9-c84b0190fa84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf\" (UID: \"39938499-3689-4d6f-8ce9-c84b0190fa84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf" Sep 11 00:07:14.820970 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:14.820891 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf" Sep 11 00:07:15.109028 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:15.109005 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf"] Sep 11 00:07:15.111018 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:07:15.110990 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39938499_3689_4d6f_8ce9_c84b0190fa84.slice/crio-0e6f98f04a10dad8bb05978731a4e444261ccc03cb7d188994a0e18d63a07d94 WatchSource:0}: Error finding container 0e6f98f04a10dad8bb05978731a4e444261ccc03cb7d188994a0e18d63a07d94: Status 404 returned error can't find the container with id 0e6f98f04a10dad8bb05978731a4e444261ccc03cb7d188994a0e18d63a07d94 Sep 11 00:07:15.241731 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:15.241700 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf" event={"ID":"39938499-3689-4d6f-8ce9-c84b0190fa84","Type":"ContainerStarted","Data":"0e6f98f04a10dad8bb05978731a4e444261ccc03cb7d188994a0e18d63a07d94"} Sep 11 00:07:16.247688 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:16.247654 2583 generic.go:356] "Generic (PLEG): container finished" podID="39938499-3689-4d6f-8ce9-c84b0190fa84" containerID="9cd9be695e3cd728c8f0610386c58c700dc4e7a4b0cf9a82cd0193526effc31f" exitCode=0 Sep 11 00:07:16.248108 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:16.247736 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf" event={"ID":"39938499-3689-4d6f-8ce9-c84b0190fa84","Type":"ContainerDied","Data":"9cd9be695e3cd728c8f0610386c58c700dc4e7a4b0cf9a82cd0193526effc31f"} Sep 11 00:07:17.377290 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:17.377267 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf" Sep 11 00:07:17.439748 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:17.439718 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39938499-3689-4d6f-8ce9-c84b0190fa84-maas-api-service-ca\") pod \"39938499-3689-4d6f-8ce9-c84b0190fa84\" (UID: \"39938499-3689-4d6f-8ce9-c84b0190fa84\") " Sep 11 00:07:17.440097 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:17.440041 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39938499-3689-4d6f-8ce9-c84b0190fa84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "39938499-3689-4d6f-8ce9-c84b0190fa84" (UID: "39938499-3689-4d6f-8ce9-c84b0190fa84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:07:17.541290 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:17.541210 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39938499-3689-4d6f-8ce9-c84b0190fa84-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:18.259051 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:18.259008 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf" event={"ID":"39938499-3689-4d6f-8ce9-c84b0190fa84","Type":"ContainerDied","Data":"0e6f98f04a10dad8bb05978731a4e444261ccc03cb7d188994a0e18d63a07d94"} Sep 11 00:07:18.259234 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:18.259065 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e6f98f04a10dad8bb05978731a4e444261ccc03cb7d188994a0e18d63a07d94" Sep 11 00:07:18.259234 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:18.259022 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf" Sep 11 00:07:19.266340 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.266310 2583 generic.go:356] "Generic (PLEG): container finished" podID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerID="acd4f87379a1439b03b4ece29e42615e2a2fc056ffb49c277ec02a0d317f179e" exitCode=0 Sep 11 00:07:19.266775 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.266396 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" event={"ID":"c1dbf00a-72c9-41c5-a00d-657096586bb4","Type":"ContainerDied","Data":"acd4f87379a1439b03b4ece29e42615e2a2fc056ffb49c277ec02a0d317f179e"} Sep 11 00:07:19.266775 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.266479 2583 scope.go:117] "RemoveContainer" containerID="e520c1cd09e1fd735bd156ec1a698b188bf86798f7a2c213cef105c31e346ab1" Sep 11 00:07:19.345761 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.345739 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:07:19.455967 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.455889 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-workload-certs\") pod \"c1dbf00a-72c9-41c5-a00d-657096586bb4\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " Sep 11 00:07:19.455967 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.455936 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-credential-socket\") pod \"c1dbf00a-72c9-41c5-a00d-657096586bb4\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " Sep 11 00:07:19.456223 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.455978 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-envoy\") pod \"c1dbf00a-72c9-41c5-a00d-657096586bb4\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " Sep 11 00:07:19.456223 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.455997 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dbf00a-72c9-41c5-a00d-657096586bb4-istiod-ca-cert\") pod \"c1dbf00a-72c9-41c5-a00d-657096586bb4\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " Sep 11 00:07:19.456223 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456021 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-token\") pod \"c1dbf00a-72c9-41c5-a00d-657096586bb4\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " Sep 11 00:07:19.456223 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456064 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmjp6\" (UniqueName: \"kubernetes.io/projected/c1dbf00a-72c9-41c5-a00d-657096586bb4-kube-api-access-jmjp6\") pod \"c1dbf00a-72c9-41c5-a00d-657096586bb4\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " Sep 11 00:07:19.456223 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456106 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-workload-socket\") pod \"c1dbf00a-72c9-41c5-a00d-657096586bb4\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " Sep 11 00:07:19.456223 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456153 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-data\") pod \"c1dbf00a-72c9-41c5-a00d-657096586bb4\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " Sep 11 00:07:19.456523 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456242 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c1dbf00a-72c9-41c5-a00d-657096586bb4" (UID: "c1dbf00a-72c9-41c5-a00d-657096586bb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:07:19.456523 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456359 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c1dbf00a-72c9-41c5-a00d-657096586bb4" (UID: "c1dbf00a-72c9-41c5-a00d-657096586bb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:07:19.456523 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456443 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c1dbf00a-72c9-41c5-a00d-657096586bb4" (UID: "c1dbf00a-72c9-41c5-a00d-657096586bb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:07:19.456523 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456503 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-data" (OuterVolumeSpecName: "istio-data") pod "c1dbf00a-72c9-41c5-a00d-657096586bb4" (UID: "c1dbf00a-72c9-41c5-a00d-657096586bb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:07:19.456695 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456547 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-podinfo\") pod \"c1dbf00a-72c9-41c5-a00d-657096586bb4\" (UID: \"c1dbf00a-72c9-41c5-a00d-657096586bb4\") " Sep 11 00:07:19.456695 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456542 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dbf00a-72c9-41c5-a00d-657096586bb4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c1dbf00a-72c9-41c5-a00d-657096586bb4" (UID: "c1dbf00a-72c9-41c5-a00d-657096586bb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:07:19.456860 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456840 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.456900 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456865 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.456900 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456879 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dbf00a-72c9-41c5-a00d-657096586bb4-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.456900 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456893 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.456995 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.456906 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.458282 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.458244 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-token" (OuterVolumeSpecName: "istio-token") pod "c1dbf00a-72c9-41c5-a00d-657096586bb4" (UID: "c1dbf00a-72c9-41c5-a00d-657096586bb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:07:19.458282 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.458258 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c1dbf00a-72c9-41c5-a00d-657096586bb4" (UID: "c1dbf00a-72c9-41c5-a00d-657096586bb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:07:19.458421 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.458298 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dbf00a-72c9-41c5-a00d-657096586bb4-kube-api-access-jmjp6" (OuterVolumeSpecName: "kube-api-access-jmjp6") pod "c1dbf00a-72c9-41c5-a00d-657096586bb4" (UID: "c1dbf00a-72c9-41c5-a00d-657096586bb4"). InnerVolumeSpecName "kube-api-access-jmjp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:07:19.458478 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.458459 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c1dbf00a-72c9-41c5-a00d-657096586bb4" (UID: "c1dbf00a-72c9-41c5-a00d-657096586bb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:07:19.557787 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.557749 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.557787 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.557782 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.557787 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.557792 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmjp6\" (UniqueName: \"kubernetes.io/projected/c1dbf00a-72c9-41c5-a00d-657096586bb4-kube-api-access-jmjp6\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.557787 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:19.557801 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dbf00a-72c9-41c5-a00d-657096586bb4-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:20.272192 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:20.272159 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" Sep 11 00:07:20.272192 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:20.272168 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58" event={"ID":"c1dbf00a-72c9-41c5-a00d-657096586bb4","Type":"ContainerDied","Data":"4499cb62635ce6c016913d66d514ddc60c9495eec05de9703dbd32f2f22a61b2"} Sep 11 00:07:20.272672 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:20.272227 2583 scope.go:117] "RemoveContainer" containerID="acd4f87379a1439b03b4ece29e42615e2a2fc056ffb49c277ec02a0d317f179e" Sep 11 00:07:20.295667 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:20.295596 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58"] Sep 11 00:07:20.301629 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:20.301608 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c3755-openshift-default-8554947db6-xqb58"] Sep 11 00:07:21.306990 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:21.306950 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" path="/var/lib/kubelet/pods/c1dbf00a-72c9-41c5-a00d-657096586bb4/volumes" Sep 11 00:07:25.046433 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.046398 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp"] Sep 11 00:07:25.046890 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.046615 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" podUID="2a347ab7-4db8-4712-9ddb-787b730efe86" containerName="payload-pre-processing" containerID="cri-o://d961cd4971bd9b30f84b642f6145fda88eb7560d5563fc5dd279cd2446b44dcd" gracePeriod=30 Sep 11 00:07:25.228144 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.228109 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf"] Sep 11 00:07:25.230125 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.230099 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-da6814-nsqkf"] Sep 11 00:07:25.291484 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.291450 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" Sep 11 00:07:25.299983 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.299916 2583 generic.go:356] "Generic (PLEG): container finished" podID="2a347ab7-4db8-4712-9ddb-787b730efe86" containerID="d961cd4971bd9b30f84b642f6145fda88eb7560d5563fc5dd279cd2446b44dcd" exitCode=0 Sep 11 00:07:25.299983 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.299960 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" Sep 11 00:07:25.300143 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.299958 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" event={"ID":"2a347ab7-4db8-4712-9ddb-787b730efe86","Type":"ContainerDied","Data":"d961cd4971bd9b30f84b642f6145fda88eb7560d5563fc5dd279cd2446b44dcd"} Sep 11 00:07:25.300143 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.300116 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp" event={"ID":"2a347ab7-4db8-4712-9ddb-787b730efe86","Type":"ContainerDied","Data":"c4ce8ca346e402cbaa31d94aa62635507c37b0b7cf177af5ed44005d79d2bf66"} Sep 11 00:07:25.300227 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.300144 2583 scope.go:117] "RemoveContainer" containerID="d961cd4971bd9b30f84b642f6145fda88eb7560d5563fc5dd279cd2446b44dcd" Sep 11 00:07:25.308349 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.308322 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39938499-3689-4d6f-8ce9-c84b0190fa84" path="/var/lib/kubelet/pods/39938499-3689-4d6f-8ce9-c84b0190fa84/volumes" Sep 11 00:07:25.309323 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.309301 2583 scope.go:117] "RemoveContainer" containerID="d961cd4971bd9b30f84b642f6145fda88eb7560d5563fc5dd279cd2446b44dcd" Sep 11 00:07:25.309593 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:07:25.309571 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d961cd4971bd9b30f84b642f6145fda88eb7560d5563fc5dd279cd2446b44dcd\": container with ID starting with d961cd4971bd9b30f84b642f6145fda88eb7560d5563fc5dd279cd2446b44dcd not found: ID does not exist" containerID="d961cd4971bd9b30f84b642f6145fda88eb7560d5563fc5dd279cd2446b44dcd" Sep 11 00:07:25.309683 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.309599 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d961cd4971bd9b30f84b642f6145fda88eb7560d5563fc5dd279cd2446b44dcd"} err="failed to get container status \"d961cd4971bd9b30f84b642f6145fda88eb7560d5563fc5dd279cd2446b44dcd\": rpc error: code = NotFound desc = could not find container \"d961cd4971bd9b30f84b642f6145fda88eb7560d5563fc5dd279cd2446b44dcd\": container with ID starting with d961cd4971bd9b30f84b642f6145fda88eb7560d5563fc5dd279cd2446b44dcd not found: ID does not exist" Sep 11 00:07:25.405198 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.405167 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6pfh\" (UniqueName: \"kubernetes.io/projected/2a347ab7-4db8-4712-9ddb-787b730efe86-kube-api-access-p6pfh\") pod \"2a347ab7-4db8-4712-9ddb-787b730efe86\" (UID: \"2a347ab7-4db8-4712-9ddb-787b730efe86\") " Sep 11 00:07:25.405364 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.405254 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a347ab7-4db8-4712-9ddb-787b730efe86-plugins-config-volume\") pod \"2a347ab7-4db8-4712-9ddb-787b730efe86\" (UID: \"2a347ab7-4db8-4712-9ddb-787b730efe86\") " Sep 11 00:07:25.405730 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.405696 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a347ab7-4db8-4712-9ddb-787b730efe86-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a347ab7-4db8-4712-9ddb-787b730efe86" (UID: "2a347ab7-4db8-4712-9ddb-787b730efe86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:07:25.407370 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.407349 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a347ab7-4db8-4712-9ddb-787b730efe86-kube-api-access-p6pfh" (OuterVolumeSpecName: "kube-api-access-p6pfh") pod "2a347ab7-4db8-4712-9ddb-787b730efe86" (UID: "2a347ab7-4db8-4712-9ddb-787b730efe86"). InnerVolumeSpecName "kube-api-access-p6pfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:07:25.506692 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.506660 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a347ab7-4db8-4712-9ddb-787b730efe86-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:25.506692 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.506688 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6pfh\" (UniqueName: \"kubernetes.io/projected/2a347ab7-4db8-4712-9ddb-787b730efe86-kube-api-access-p6pfh\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:25.624898 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.624855 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp"] Sep 11 00:07:25.626918 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:25.626897 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-da6814-7b8b8b9466-t4kdp"] Sep 11 00:07:27.308562 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:27.308520 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a347ab7-4db8-4712-9ddb-787b730efe86" path="/var/lib/kubelet/pods/2a347ab7-4db8-4712-9ddb-787b730efe86/volumes" Sep 11 00:07:31.987359 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.987317 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98"] Sep 11 00:07:31.987742 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.987656 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerName="istio-proxy" Sep 11 00:07:31.987742 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.987668 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerName="istio-proxy" Sep 11 00:07:31.987742 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.987680 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a347ab7-4db8-4712-9ddb-787b730efe86" containerName="payload-pre-processing" Sep 11 00:07:31.987742 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.987686 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a347ab7-4db8-4712-9ddb-787b730efe86" containerName="payload-pre-processing" Sep 11 00:07:31.987742 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.987696 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39938499-3689-4d6f-8ce9-c84b0190fa84" containerName="revoke-keys" Sep 11 00:07:31.987742 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.987701 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="39938499-3689-4d6f-8ce9-c84b0190fa84" containerName="revoke-keys" Sep 11 00:07:31.987931 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.987769 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerName="istio-proxy" Sep 11 00:07:31.987931 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.987778 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="39938499-3689-4d6f-8ce9-c84b0190fa84" containerName="revoke-keys" Sep 11 00:07:31.987931 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.987784 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a347ab7-4db8-4712-9ddb-787b730efe86" containerName="payload-pre-processing" Sep 11 00:07:31.991903 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.991886 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98" Sep 11 00:07:31.994171 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.994149 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:07:31.994171 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.994163 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 11 00:07:31.997517 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:31.997494 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98"] Sep 11 00:07:32.165708 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:32.165672 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8413c96d-2808-48f6-8cb5-7ffc3d98ebc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98\" (UID: \"8413c96d-2808-48f6-8cb5-7ffc3d98ebc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98" Sep 11 00:07:32.266773 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:32.266683 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8413c96d-2808-48f6-8cb5-7ffc3d98ebc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98\" (UID: \"8413c96d-2808-48f6-8cb5-7ffc3d98ebc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98" Sep 11 00:07:32.267323 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:32.267301 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8413c96d-2808-48f6-8cb5-7ffc3d98ebc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98\" (UID: \"8413c96d-2808-48f6-8cb5-7ffc3d98ebc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98" Sep 11 00:07:32.304113 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:32.304060 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98" Sep 11 00:07:32.572333 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:32.572278 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98"] Sep 11 00:07:32.574913 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:07:32.574881 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8413c96d_2808_48f6_8cb5_7ffc3d98ebc0.slice/crio-0a8784cb8d561d767fc2dc087ab6c873fe078f7f1d12feb22a1763942f1e0449 WatchSource:0}: Error finding container 0a8784cb8d561d767fc2dc087ab6c873fe078f7f1d12feb22a1763942f1e0449: Status 404 returned error can't find the container with id 0a8784cb8d561d767fc2dc087ab6c873fe078f7f1d12feb22a1763942f1e0449 Sep 11 00:07:33.339817 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:33.339784 2583 generic.go:356] "Generic (PLEG): container finished" podID="8413c96d-2808-48f6-8cb5-7ffc3d98ebc0" containerID="0e1237f9a1197b723e75fe87e6a63781848f488f5ed41c88a4267c25fb6b5013" exitCode=0 Sep 11 00:07:33.340204 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:33.339866 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98" event={"ID":"8413c96d-2808-48f6-8cb5-7ffc3d98ebc0","Type":"ContainerDied","Data":"0e1237f9a1197b723e75fe87e6a63781848f488f5ed41c88a4267c25fb6b5013"} Sep 11 00:07:33.340204 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:33.339921 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98" event={"ID":"8413c96d-2808-48f6-8cb5-7ffc3d98ebc0","Type":"ContainerStarted","Data":"0a8784cb8d561d767fc2dc087ab6c873fe078f7f1d12feb22a1763942f1e0449"} Sep 11 00:07:33.556567 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:33.556474 2583 scope.go:117] "RemoveContainer" containerID="dcedd1b5e3cb27e24217458c38818088b11b96a2bfb3326fa351238ce01f55e9" Sep 11 00:07:33.564727 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:33.564700 2583 scope.go:117] "RemoveContainer" containerID="a0990b3d9298b745101c2302c6fa80ec46d601b2c81c7e622a49469f65bb3466" Sep 11 00:07:34.477672 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:34.477645 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98" Sep 11 00:07:34.588326 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:34.588295 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8413c96d-2808-48f6-8cb5-7ffc3d98ebc0-maas-api-service-ca\") pod \"8413c96d-2808-48f6-8cb5-7ffc3d98ebc0\" (UID: \"8413c96d-2808-48f6-8cb5-7ffc3d98ebc0\") " Sep 11 00:07:34.588648 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:34.588623 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8413c96d-2808-48f6-8cb5-7ffc3d98ebc0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8413c96d-2808-48f6-8cb5-7ffc3d98ebc0" (UID: "8413c96d-2808-48f6-8cb5-7ffc3d98ebc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:07:34.689223 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:34.689185 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8413c96d-2808-48f6-8cb5-7ffc3d98ebc0-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:35.350473 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:35.350448 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98" Sep 11 00:07:35.350637 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:35.350486 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98" event={"ID":"8413c96d-2808-48f6-8cb5-7ffc3d98ebc0","Type":"ContainerDied","Data":"0a8784cb8d561d767fc2dc087ab6c873fe078f7f1d12feb22a1763942f1e0449"} Sep 11 00:07:35.350637 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:35.350536 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a8784cb8d561d767fc2dc087ab6c873fe078f7f1d12feb22a1763942f1e0449" Sep 11 00:07:42.541040 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:42.541003 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr"] Sep 11 00:07:42.541535 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:42.541251 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" podUID="e9a63268-3b36-46a6-86d4-62efd768c9da" containerName="payload-pre-processing" containerID="cri-o://e7d1d153d9e6801281e42495a8b5431ac1aa6353b049674f883c0f6f12a8e498" gracePeriod=30 Sep 11 00:07:42.730575 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:42.730538 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98"] Sep 11 00:07:42.733463 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:42.733425 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6fa41e-ktt98"] Sep 11 00:07:42.786305 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:42.786230 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" Sep 11 00:07:42.862795 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:42.862766 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5tg5\" (UniqueName: \"kubernetes.io/projected/e9a63268-3b36-46a6-86d4-62efd768c9da-kube-api-access-r5tg5\") pod \"e9a63268-3b36-46a6-86d4-62efd768c9da\" (UID: \"e9a63268-3b36-46a6-86d4-62efd768c9da\") " Sep 11 00:07:42.863007 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:42.862843 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a63268-3b36-46a6-86d4-62efd768c9da-plugins-config-volume\") pod \"e9a63268-3b36-46a6-86d4-62efd768c9da\" (UID: \"e9a63268-3b36-46a6-86d4-62efd768c9da\") " Sep 11 00:07:42.863283 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:42.863257 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a63268-3b36-46a6-86d4-62efd768c9da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9a63268-3b36-46a6-86d4-62efd768c9da" (UID: "e9a63268-3b36-46a6-86d4-62efd768c9da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:07:42.864884 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:42.864855 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a63268-3b36-46a6-86d4-62efd768c9da-kube-api-access-r5tg5" (OuterVolumeSpecName: "kube-api-access-r5tg5") pod "e9a63268-3b36-46a6-86d4-62efd768c9da" (UID: "e9a63268-3b36-46a6-86d4-62efd768c9da"). InnerVolumeSpecName "kube-api-access-r5tg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:07:42.964314 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:42.964272 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a63268-3b36-46a6-86d4-62efd768c9da-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:42.964314 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:42.964305 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5tg5\" (UniqueName: \"kubernetes.io/projected/e9a63268-3b36-46a6-86d4-62efd768c9da-kube-api-access-r5tg5\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:43.312231 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:43.312135 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8413c96d-2808-48f6-8cb5-7ffc3d98ebc0" path="/var/lib/kubelet/pods/8413c96d-2808-48f6-8cb5-7ffc3d98ebc0/volumes" Sep 11 00:07:43.389673 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:43.389632 2583 generic.go:356] "Generic (PLEG): container finished" podID="e9a63268-3b36-46a6-86d4-62efd768c9da" containerID="e7d1d153d9e6801281e42495a8b5431ac1aa6353b049674f883c0f6f12a8e498" exitCode=0 Sep 11 00:07:43.389857 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:43.389697 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" Sep 11 00:07:43.389857 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:43.389718 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" event={"ID":"e9a63268-3b36-46a6-86d4-62efd768c9da","Type":"ContainerDied","Data":"e7d1d153d9e6801281e42495a8b5431ac1aa6353b049674f883c0f6f12a8e498"} Sep 11 00:07:43.389857 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:43.389778 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr" event={"ID":"e9a63268-3b36-46a6-86d4-62efd768c9da","Type":"ContainerDied","Data":"86a92ce3de910d6811579728d6a65e6e8752a7641c21d98b64c3215e3dbb86d1"} Sep 11 00:07:43.389857 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:43.389803 2583 scope.go:117] "RemoveContainer" containerID="e7d1d153d9e6801281e42495a8b5431ac1aa6353b049674f883c0f6f12a8e498" Sep 11 00:07:43.399630 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:43.399609 2583 scope.go:117] "RemoveContainer" containerID="e7d1d153d9e6801281e42495a8b5431ac1aa6353b049674f883c0f6f12a8e498" Sep 11 00:07:43.399909 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:07:43.399887 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d1d153d9e6801281e42495a8b5431ac1aa6353b049674f883c0f6f12a8e498\": container with ID starting with e7d1d153d9e6801281e42495a8b5431ac1aa6353b049674f883c0f6f12a8e498 not found: ID does not exist" containerID="e7d1d153d9e6801281e42495a8b5431ac1aa6353b049674f883c0f6f12a8e498" Sep 11 00:07:43.399977 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:43.399922 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d1d153d9e6801281e42495a8b5431ac1aa6353b049674f883c0f6f12a8e498"} err="failed to get container status \"e7d1d153d9e6801281e42495a8b5431ac1aa6353b049674f883c0f6f12a8e498\": rpc error: code = NotFound desc = could not find container \"e7d1d153d9e6801281e42495a8b5431ac1aa6353b049674f883c0f6f12a8e498\": container with ID starting with e7d1d153d9e6801281e42495a8b5431ac1aa6353b049674f883c0f6f12a8e498 not found: ID does not exist" Sep 11 00:07:43.408085 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:43.408047 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr"] Sep 11 00:07:43.411699 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:43.411677 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6fa41e-6d96f4dd4b-bgvtr"] Sep 11 00:07:45.307953 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:45.307920 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9a63268-3b36-46a6-86d4-62efd768c9da" path="/var/lib/kubelet/pods/e9a63268-3b36-46a6-86d4-62efd768c9da/volumes" Sep 11 00:07:49.716886 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.716849 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv"] Sep 11 00:07:49.717368 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.717326 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a63268-3b36-46a6-86d4-62efd768c9da" containerName="payload-pre-processing" Sep 11 00:07:49.717368 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.717347 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a63268-3b36-46a6-86d4-62efd768c9da" containerName="payload-pre-processing" Sep 11 00:07:49.717368 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.717366 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerName="istio-proxy" Sep 11 00:07:49.717543 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.717374 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerName="istio-proxy" Sep 11 00:07:49.717543 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.717404 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8413c96d-2808-48f6-8cb5-7ffc3d98ebc0" containerName="revoke-keys" Sep 11 00:07:49.717543 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.717414 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8413c96d-2808-48f6-8cb5-7ffc3d98ebc0" containerName="revoke-keys" Sep 11 00:07:49.717543 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.717487 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8413c96d-2808-48f6-8cb5-7ffc3d98ebc0" containerName="revoke-keys" Sep 11 00:07:49.717543 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.717500 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a63268-3b36-46a6-86d4-62efd768c9da" containerName="payload-pre-processing" Sep 11 00:07:49.717543 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.717509 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1dbf00a-72c9-41c5-a00d-657096586bb4" containerName="istio-proxy" Sep 11 00:07:49.721936 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.721915 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv" Sep 11 00:07:49.724575 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.724556 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 11 00:07:49.724688 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.724631 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:07:49.728104 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.728084 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv"] Sep 11 00:07:49.820145 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.820112 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50f7d3a-612c-4755-a17b-2bda236a0471-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv\" (UID: \"a50f7d3a-612c-4755-a17b-2bda236a0471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv" Sep 11 00:07:49.920757 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.920731 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50f7d3a-612c-4755-a17b-2bda236a0471-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv\" (UID: \"a50f7d3a-612c-4755-a17b-2bda236a0471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv" Sep 11 00:07:49.921338 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:49.921319 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50f7d3a-612c-4755-a17b-2bda236a0471-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv\" (UID: \"a50f7d3a-612c-4755-a17b-2bda236a0471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv" Sep 11 00:07:50.033741 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:50.033652 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv" Sep 11 00:07:50.301843 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:50.301763 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv"] Sep 11 00:07:50.304780 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:07:50.304755 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50f7d3a_612c_4755_a17b_2bda236a0471.slice/crio-3649b60bd97b93dbcc041f087dc118c4460645a482b334e1267586e753d540f0 WatchSource:0}: Error finding container 3649b60bd97b93dbcc041f087dc118c4460645a482b334e1267586e753d540f0: Status 404 returned error can't find the container with id 3649b60bd97b93dbcc041f087dc118c4460645a482b334e1267586e753d540f0 Sep 11 00:07:50.424132 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:50.424093 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv" event={"ID":"a50f7d3a-612c-4755-a17b-2bda236a0471","Type":"ContainerStarted","Data":"3649b60bd97b93dbcc041f087dc118c4460645a482b334e1267586e753d540f0"} Sep 11 00:07:51.430368 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:51.430336 2583 generic.go:356] "Generic (PLEG): container finished" podID="a50f7d3a-612c-4755-a17b-2bda236a0471" containerID="03bc4059944a030e4ef75783128da4b805d125a5dc6c9d882692510ad7a256ef" exitCode=0 Sep 11 00:07:51.430813 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:51.430407 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv" event={"ID":"a50f7d3a-612c-4755-a17b-2bda236a0471","Type":"ContainerDied","Data":"03bc4059944a030e4ef75783128da4b805d125a5dc6c9d882692510ad7a256ef"} Sep 11 00:07:52.565570 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:52.565545 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv" Sep 11 00:07:52.643124 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:52.643093 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50f7d3a-612c-4755-a17b-2bda236a0471-maas-api-service-ca\") pod \"a50f7d3a-612c-4755-a17b-2bda236a0471\" (UID: \"a50f7d3a-612c-4755-a17b-2bda236a0471\") " Sep 11 00:07:52.643458 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:52.643435 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50f7d3a-612c-4755-a17b-2bda236a0471-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a50f7d3a-612c-4755-a17b-2bda236a0471" (UID: "a50f7d3a-612c-4755-a17b-2bda236a0471"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:07:52.744533 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:52.744459 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50f7d3a-612c-4755-a17b-2bda236a0471-maas-api-service-ca\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:07:53.440193 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:53.440164 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv" Sep 11 00:07:53.440375 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:53.440197 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv" event={"ID":"a50f7d3a-612c-4755-a17b-2bda236a0471","Type":"ContainerDied","Data":"3649b60bd97b93dbcc041f087dc118c4460645a482b334e1267586e753d540f0"} Sep 11 00:07:53.440375 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:07:53.440248 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3649b60bd97b93dbcc041f087dc118c4460645a482b334e1267586e753d540f0" Sep 11 00:08:00.236053 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.235963 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6"] Sep 11 00:08:00.236549 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.236244 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" podUID="ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846" containerName="payload-processing" containerID="cri-o://f05478f5cd6745813fb5f7fb08556709cd03e804d8eb7e4977edbb92fbc0fe42" gracePeriod=30 Sep 11 00:08:00.246565 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.246539 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd"] Sep 11 00:08:00.246808 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.246755 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" podUID="bd428f77-aecd-4b65-9770-577af9e103a3" containerName="payload-pre-processing" containerID="cri-o://a975bd82f904101c4e6e522fc45de45fe74de7144704b2bc0858a437eba267a4" gracePeriod=30 Sep 11 00:08:00.403987 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.403953 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv"] Sep 11 00:08:00.407456 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.407422 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-37c598-b69zv"] Sep 11 00:08:00.476799 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.476768 2583 generic.go:356] "Generic (PLEG): container finished" podID="ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846" containerID="f05478f5cd6745813fb5f7fb08556709cd03e804d8eb7e4977edbb92fbc0fe42" exitCode=0 Sep 11 00:08:00.476921 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.476840 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" event={"ID":"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846","Type":"ContainerDied","Data":"f05478f5cd6745813fb5f7fb08556709cd03e804d8eb7e4977edbb92fbc0fe42"} Sep 11 00:08:00.476921 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.476903 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" event={"ID":"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846","Type":"ContainerDied","Data":"0bcc07c0598d4b0a4b0ba2543d7343bcc71c49a22208cd18596c3ca9c2d639a7"} Sep 11 00:08:00.477023 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.476928 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bcc07c0598d4b0a4b0ba2543d7343bcc71c49a22208cd18596c3ca9c2d639a7" Sep 11 00:08:00.478721 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.478702 2583 generic.go:356] "Generic (PLEG): container finished" podID="bd428f77-aecd-4b65-9770-577af9e103a3" containerID="a975bd82f904101c4e6e522fc45de45fe74de7144704b2bc0858a437eba267a4" exitCode=0 Sep 11 00:08:00.478834 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.478739 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" event={"ID":"bd428f77-aecd-4b65-9770-577af9e103a3","Type":"ContainerDied","Data":"a975bd82f904101c4e6e522fc45de45fe74de7144704b2bc0858a437eba267a4"} Sep 11 00:08:00.487138 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.487088 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" Sep 11 00:08:00.516634 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.511401 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846-plugins-config-volume\") pod \"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846\" (UID: \"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846\") " Sep 11 00:08:00.516634 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.511518 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsxj8\" (UniqueName: \"kubernetes.io/projected/ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846-kube-api-access-lsxj8\") pod \"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846\" (UID: \"ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846\") " Sep 11 00:08:00.516634 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.513208 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846" (UID: "ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:08:00.517798 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.517773 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846-kube-api-access-lsxj8" (OuterVolumeSpecName: "kube-api-access-lsxj8") pod "ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846" (UID: "ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846"). InnerVolumeSpecName "kube-api-access-lsxj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:08:00.520743 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.520723 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" Sep 11 00:08:00.612932 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.612897 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2mlg\" (UniqueName: \"kubernetes.io/projected/bd428f77-aecd-4b65-9770-577af9e103a3-kube-api-access-l2mlg\") pod \"bd428f77-aecd-4b65-9770-577af9e103a3\" (UID: \"bd428f77-aecd-4b65-9770-577af9e103a3\") " Sep 11 00:08:00.613139 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.612956 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd428f77-aecd-4b65-9770-577af9e103a3-plugins-config-volume\") pod \"bd428f77-aecd-4b65-9770-577af9e103a3\" (UID: \"bd428f77-aecd-4b65-9770-577af9e103a3\") " Sep 11 00:08:00.613139 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.613111 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:08:00.613139 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.613125 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsxj8\" (UniqueName: \"kubernetes.io/projected/ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846-kube-api-access-lsxj8\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:08:00.613458 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.613432 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd428f77-aecd-4b65-9770-577af9e103a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd428f77-aecd-4b65-9770-577af9e103a3" (UID: "bd428f77-aecd-4b65-9770-577af9e103a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:08:00.614953 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.614932 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd428f77-aecd-4b65-9770-577af9e103a3-kube-api-access-l2mlg" (OuterVolumeSpecName: "kube-api-access-l2mlg") pod "bd428f77-aecd-4b65-9770-577af9e103a3" (UID: "bd428f77-aecd-4b65-9770-577af9e103a3"). InnerVolumeSpecName "kube-api-access-l2mlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:08:00.713969 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.713937 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2mlg\" (UniqueName: \"kubernetes.io/projected/bd428f77-aecd-4b65-9770-577af9e103a3-kube-api-access-l2mlg\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:08:00.713969 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:00.713961 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd428f77-aecd-4b65-9770-577af9e103a3-plugins-config-volume\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:08:01.307833 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:01.307795 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a50f7d3a-612c-4755-a17b-2bda236a0471" path="/var/lib/kubelet/pods/a50f7d3a-612c-4755-a17b-2bda236a0471/volumes" Sep 11 00:08:01.484977 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:01.484941 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" Sep 11 00:08:01.485200 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:01.484994 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd" event={"ID":"bd428f77-aecd-4b65-9770-577af9e103a3","Type":"ContainerDied","Data":"ad73b943763ea306bdcc9409edc0b78c91ae8a3b634016f1bececd2b52825ccc"} Sep 11 00:08:01.485200 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:01.485053 2583 scope.go:117] "RemoveContainer" containerID="a975bd82f904101c4e6e522fc45de45fe74de7144704b2bc0858a437eba267a4" Sep 11 00:08:01.485200 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:01.485130 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6" Sep 11 00:08:01.512468 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:01.512398 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6"] Sep 11 00:08:01.516462 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:01.516438 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-37c598-6b558895f5-ht2d6"] Sep 11 00:08:01.526558 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:01.526523 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd"] Sep 11 00:08:01.530372 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:01.530347 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-37c598-5cfc74bbf4-8mdpd"] Sep 11 00:08:03.307641 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:03.307605 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd428f77-aecd-4b65-9770-577af9e103a3" path="/var/lib/kubelet/pods/bd428f77-aecd-4b65-9770-577af9e103a3/volumes" Sep 11 00:08:03.308018 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:03.307952 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846" path="/var/lib/kubelet/pods/ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846/volumes" Sep 11 00:08:07.043520 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:07.043480 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz"] Sep 11 00:08:07.043973 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:07.043722 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerName="istio-proxy" containerID="cri-o://738f52849bdc5dc1aa6cb673e634fb605bf98881b8cb5d1393ff6bae4b2688b5" gracePeriod=30 Sep 11 00:08:12.299957 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.299932 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 11 00:08:12.417645 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.417611 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5chvd\" (UniqueName: \"kubernetes.io/projected/f28b6070-193d-4086-ba4c-b679fd2c84d4-kube-api-access-5chvd\") pod \"f28b6070-193d-4086-ba4c-b679fd2c84d4\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " Sep 11 00:08:12.417811 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.417653 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-workload-certs\") pod \"f28b6070-193d-4086-ba4c-b679fd2c84d4\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " Sep 11 00:08:12.417811 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.417693 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-podinfo\") pod \"f28b6070-193d-4086-ba4c-b679fd2c84d4\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " Sep 11 00:08:12.417811 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.417718 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-token\") pod \"f28b6070-193d-4086-ba4c-b679fd2c84d4\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " Sep 11 00:08:12.417997 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.417832 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-data\") pod \"f28b6070-193d-4086-ba4c-b679fd2c84d4\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " Sep 11 00:08:12.417997 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.417877 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28b6070-193d-4086-ba4c-b679fd2c84d4-istiod-ca-cert\") pod \"f28b6070-193d-4086-ba4c-b679fd2c84d4\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " Sep 11 00:08:12.417997 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.417902 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-credential-socket\") pod \"f28b6070-193d-4086-ba4c-b679fd2c84d4\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " Sep 11 00:08:12.417997 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.417975 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-workload-socket\") pod \"f28b6070-193d-4086-ba4c-b679fd2c84d4\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " Sep 11 00:08:12.418242 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.418010 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-envoy\") pod \"f28b6070-193d-4086-ba4c-b679fd2c84d4\" (UID: \"f28b6070-193d-4086-ba4c-b679fd2c84d4\") " Sep 11 00:08:12.418242 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.418053 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f28b6070-193d-4086-ba4c-b679fd2c84d4" (UID: "f28b6070-193d-4086-ba4c-b679fd2c84d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:08:12.418242 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.418217 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f28b6070-193d-4086-ba4c-b679fd2c84d4" (UID: "f28b6070-193d-4086-ba4c-b679fd2c84d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:08:12.418396 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.418292 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f28b6070-193d-4086-ba4c-b679fd2c84d4" (UID: "f28b6070-193d-4086-ba4c-b679fd2c84d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:08:12.418396 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.418312 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28b6070-193d-4086-ba4c-b679fd2c84d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f28b6070-193d-4086-ba4c-b679fd2c84d4" (UID: "f28b6070-193d-4086-ba4c-b679fd2c84d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:08:12.418396 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.418370 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-workload-certs\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:08:12.418396 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.418389 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-credential-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:08:12.418666 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.418631 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "f28b6070-193d-4086-ba4c-b679fd2c84d4" (UID: "f28b6070-193d-4086-ba4c-b679fd2c84d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:08:12.420117 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.420081 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f28b6070-193d-4086-ba4c-b679fd2c84d4" (UID: "f28b6070-193d-4086-ba4c-b679fd2c84d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:08:12.420245 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.420161 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f28b6070-193d-4086-ba4c-b679fd2c84d4" (UID: "f28b6070-193d-4086-ba4c-b679fd2c84d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:08:12.420245 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.420206 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "f28b6070-193d-4086-ba4c-b679fd2c84d4" (UID: "f28b6070-193d-4086-ba4c-b679fd2c84d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:08:12.420359 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.420271 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28b6070-193d-4086-ba4c-b679fd2c84d4-kube-api-access-5chvd" (OuterVolumeSpecName: "kube-api-access-5chvd") pod "f28b6070-193d-4086-ba4c-b679fd2c84d4" (UID: "f28b6070-193d-4086-ba4c-b679fd2c84d4"). InnerVolumeSpecName "kube-api-access-5chvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:08:12.519038 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.518996 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-data\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:08:12.519038 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.519032 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28b6070-193d-4086-ba4c-b679fd2c84d4-istiod-ca-cert\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:08:12.519038 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.519041 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-workload-socket\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:08:12.519038 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.519049 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-envoy\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:08:12.519338 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.519059 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5chvd\" (UniqueName: \"kubernetes.io/projected/f28b6070-193d-4086-ba4c-b679fd2c84d4-kube-api-access-5chvd\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:08:12.519338 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.519099 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-podinfo\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:08:12.519338 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.519107 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28b6070-193d-4086-ba4c-b679fd2c84d4-istio-token\") on node \"ip-10-0-134-90.ec2.internal\" DevicePath \"\"" Sep 11 00:08:12.539384 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.539349 2583 generic.go:356] "Generic (PLEG): container finished" podID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerID="738f52849bdc5dc1aa6cb673e634fb605bf98881b8cb5d1393ff6bae4b2688b5" exitCode=0 Sep 11 00:08:12.539508 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.539404 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" Sep 11 00:08:12.539508 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.539432 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" event={"ID":"f28b6070-193d-4086-ba4c-b679fd2c84d4","Type":"ContainerDied","Data":"738f52849bdc5dc1aa6cb673e634fb605bf98881b8cb5d1393ff6bae4b2688b5"} Sep 11 00:08:12.539508 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.539487 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz" event={"ID":"f28b6070-193d-4086-ba4c-b679fd2c84d4","Type":"ContainerDied","Data":"d126f321d065f0100379f3452b41823936630839cfe018c64755113e1a690a67"} Sep 11 00:08:12.539508 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.539508 2583 scope.go:117] "RemoveContainer" containerID="738f52849bdc5dc1aa6cb673e634fb605bf98881b8cb5d1393ff6bae4b2688b5" Sep 11 00:08:12.548719 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.548695 2583 scope.go:117] "RemoveContainer" containerID="11a5428f83e80302a7644b0f448a32e5bd0ce1c6f3696e9b72c31eaca267f104" Sep 11 00:08:12.556427 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.556405 2583 scope.go:117] "RemoveContainer" containerID="738f52849bdc5dc1aa6cb673e634fb605bf98881b8cb5d1393ff6bae4b2688b5" Sep 11 00:08:12.556697 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:08:12.556677 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738f52849bdc5dc1aa6cb673e634fb605bf98881b8cb5d1393ff6bae4b2688b5\": container with ID starting with 738f52849bdc5dc1aa6cb673e634fb605bf98881b8cb5d1393ff6bae4b2688b5 not found: ID does not exist" containerID="738f52849bdc5dc1aa6cb673e634fb605bf98881b8cb5d1393ff6bae4b2688b5" Sep 11 00:08:12.556739 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.556706 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738f52849bdc5dc1aa6cb673e634fb605bf98881b8cb5d1393ff6bae4b2688b5"} err="failed to get container status \"738f52849bdc5dc1aa6cb673e634fb605bf98881b8cb5d1393ff6bae4b2688b5\": rpc error: code = NotFound desc = could not find container \"738f52849bdc5dc1aa6cb673e634fb605bf98881b8cb5d1393ff6bae4b2688b5\": container with ID starting with 738f52849bdc5dc1aa6cb673e634fb605bf98881b8cb5d1393ff6bae4b2688b5 not found: ID does not exist" Sep 11 00:08:12.556739 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.556724 2583 scope.go:117] "RemoveContainer" containerID="11a5428f83e80302a7644b0f448a32e5bd0ce1c6f3696e9b72c31eaca267f104" Sep 11 00:08:12.556991 ip-10-0-134-90 kubenswrapper[2583]: E0911 00:08:12.556972 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a5428f83e80302a7644b0f448a32e5bd0ce1c6f3696e9b72c31eaca267f104\": container with ID starting with 11a5428f83e80302a7644b0f448a32e5bd0ce1c6f3696e9b72c31eaca267f104 not found: ID does not exist" containerID="11a5428f83e80302a7644b0f448a32e5bd0ce1c6f3696e9b72c31eaca267f104" Sep 11 00:08:12.557049 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.556996 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a5428f83e80302a7644b0f448a32e5bd0ce1c6f3696e9b72c31eaca267f104"} err="failed to get container status \"11a5428f83e80302a7644b0f448a32e5bd0ce1c6f3696e9b72c31eaca267f104\": rpc error: code = NotFound desc = could not find container \"11a5428f83e80302a7644b0f448a32e5bd0ce1c6f3696e9b72c31eaca267f104\": container with ID starting with 11a5428f83e80302a7644b0f448a32e5bd0ce1c6f3696e9b72c31eaca267f104 not found: ID does not exist" Sep 11 00:08:12.559994 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.559970 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz"] Sep 11 00:08:12.567595 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:12.567567 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-37c598-openshift-default-67b54986fb-7mwqz"] Sep 11 00:08:13.308646 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:13.308609 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" path="/var/lib/kubelet/pods/f28b6070-193d-4086-ba4c-b679fd2c84d4/volumes" Sep 11 00:08:33.592920 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:33.592879 2583 scope.go:117] "RemoveContainer" containerID="93ff89558c95d1e00ae18e3295537e8531986d1832d238ae7f200074e8eef1de" Sep 11 00:08:33.600717 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:08:33.600698 2583 scope.go:117] "RemoveContainer" containerID="f05478f5cd6745813fb5f7fb08556709cd03e804d8eb7e4977edbb92fbc0fe42" Sep 11 00:12:33.649729 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:12:33.649643 2583 scope.go:117] "RemoveContainer" containerID="2592c2d645733ca7abe0a4e9357d90a08c2bd50654d9e9eebb64bde05ba01219" Sep 11 00:12:33.657550 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:12:33.657529 2583 scope.go:117] "RemoveContainer" containerID="100d887aabfce9fe08a09dbbae99d571bda8d3167ba65087b5dc33d19454d2c9" Sep 11 00:13:33.679245 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:13:33.679205 2583 scope.go:117] "RemoveContainer" containerID="0e1237f9a1197b723e75fe87e6a63781848f488f5ed41c88a4267c25fb6b5013" Sep 11 00:13:33.687024 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:13:33.687003 2583 scope.go:117] "RemoveContainer" containerID="9cd9be695e3cd728c8f0610386c58c700dc4e7a4b0cf9a82cd0193526effc31f" Sep 11 00:14:22.854512 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:22.854481 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-twk4s_7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a/server/0.log" Sep 11 00:14:23.075828 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:23.075796 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bb6f54779-5mph5_04334b39-1b36-4759-b81d-877da7cf4e37/copy-manifests/0.log" Sep 11 00:14:23.093523 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:23.093490 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bb6f54779-5mph5_04334b39-1b36-4759-b81d-877da7cf4e37/manager/0.log" Sep 11 00:14:23.880664 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:23.880635 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818080-g8d78_129166ea-7e5c-4713-832d-53942e7b36f7/cleanup/0.log" Sep 11 00:14:25.793672 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:25.793641 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn_afc3a1d4-817a-4da5-918e-12c098448c9e/istio-proxy/0.log" Sep 11 00:14:25.908454 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:25.908424 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zcqfs_73466eaa-d499-4209-be94-ba88149422b1/discovery/0.log" Sep 11 00:14:27.017885 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:27.017852 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-k9dv9_088af8e1-9142-4c01-816c-25202e79cf1b/main/0.log" Sep 11 00:14:27.366227 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:27.366189 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnz9dp_b06dac9d-4784-426d-9280-db614b0a52a8/main/0.log" Sep 11 00:14:27.495080 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:27.495050 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ns8zb_9cfacd2e-bc36-44f3-87c7-2516c3fc33c0/main/0.log" Sep 11 00:14:33.706377 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:33.706344 2583 scope.go:117] "RemoveContainer" containerID="03bc4059944a030e4ef75783128da4b805d125a5dc6c9d882692510ad7a256ef" Sep 11 00:14:35.261483 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:35.261447 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rt86f_9b40d527-4121-4aeb-8abd-c4c7c5f53688/global-pull-secret-syncer/0.log" Sep 11 00:14:35.379930 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:35.379900 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gdmff_79f4ffdc-56fe-4eb6-a9ce-a48f4d1526c9/konnectivity-agent/0.log" Sep 11 00:14:35.444742 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:35.444702 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-90.ec2.internal_8aff0765baa894ad2af911b31f796a12/haproxy/0.log" Sep 11 00:14:42.047924 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:42.047893 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bg2xd_39c93f55-cbf1-4a65-8386-dbe543395530/node-exporter/0.log" Sep 11 00:14:42.065458 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:42.065433 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bg2xd_39c93f55-cbf1-4a65-8386-dbe543395530/kube-rbac-proxy/0.log" Sep 11 00:14:42.084954 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:42.084931 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bg2xd_39c93f55-cbf1-4a65-8386-dbe543395530/init-textfile/0.log" Sep 11 00:14:45.680825 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.680782 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq"] Sep 11 00:14:45.681319 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.681216 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a50f7d3a-612c-4755-a17b-2bda236a0471" containerName="revoke-keys" Sep 11 00:14:45.681319 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.681235 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a50f7d3a-612c-4755-a17b-2bda236a0471" containerName="revoke-keys" Sep 11 00:14:45.681319 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.681249 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846" containerName="payload-processing" Sep 11 00:14:45.681319 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.681257 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846" containerName="payload-processing" Sep 11 00:14:45.681319 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.681273 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerName="istio-proxy" Sep 11 00:14:45.681319 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.681282 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerName="istio-proxy" Sep 11 00:14:45.681319 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.681298 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd428f77-aecd-4b65-9770-577af9e103a3" containerName="payload-pre-processing" Sep 11 00:14:45.681319 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.681305 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd428f77-aecd-4b65-9770-577af9e103a3" containerName="payload-pre-processing" Sep 11 00:14:45.681743 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.681384 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerName="istio-proxy" Sep 11 00:14:45.681743 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.681399 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd428f77-aecd-4b65-9770-577af9e103a3" containerName="payload-pre-processing" Sep 11 00:14:45.681743 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.681410 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed3b3fd3-2a3d-4d55-97bd-0a1b62daf846" containerName="payload-processing" Sep 11 00:14:45.681743 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.681419 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a50f7d3a-612c-4755-a17b-2bda236a0471" containerName="revoke-keys" Sep 11 00:14:45.688528 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.688498 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.691523 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.691478 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-chw9w\"/\"default-dockercfg-w5c67\"" Sep 11 00:14:45.692143 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.692122 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-chw9w\"/\"openshift-service-ca.crt\"" Sep 11 00:14:45.692262 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.692244 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-chw9w\"/\"kube-root-ca.crt\"" Sep 11 00:14:45.693421 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.693398 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq"] Sep 11 00:14:45.756693 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.756665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8p6c\" (UniqueName: \"kubernetes.io/projected/720358b7-256e-473b-8271-93bb60f8676f-kube-api-access-p8p6c\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.756868 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.756707 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/720358b7-256e-473b-8271-93bb60f8676f-proc\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.756868 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.756734 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/720358b7-256e-473b-8271-93bb60f8676f-sys\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.756868 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.756820 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/720358b7-256e-473b-8271-93bb60f8676f-podres\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.756981 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.756883 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/720358b7-256e-473b-8271-93bb60f8676f-lib-modules\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.857424 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.857392 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8p6c\" (UniqueName: \"kubernetes.io/projected/720358b7-256e-473b-8271-93bb60f8676f-kube-api-access-p8p6c\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.857616 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.857435 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/720358b7-256e-473b-8271-93bb60f8676f-proc\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.857616 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.857459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/720358b7-256e-473b-8271-93bb60f8676f-sys\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.857616 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.857489 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/720358b7-256e-473b-8271-93bb60f8676f-podres\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.857616 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.857529 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/720358b7-256e-473b-8271-93bb60f8676f-lib-modules\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.857616 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.857542 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/720358b7-256e-473b-8271-93bb60f8676f-proc\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.857616 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.857589 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/720358b7-256e-473b-8271-93bb60f8676f-sys\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.857853 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.857641 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/720358b7-256e-473b-8271-93bb60f8676f-podres\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.857853 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.857649 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/720358b7-256e-473b-8271-93bb60f8676f-lib-modules\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.865037 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.865016 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8p6c\" (UniqueName: \"kubernetes.io/projected/720358b7-256e-473b-8271-93bb60f8676f-kube-api-access-p8p6c\") pod \"perf-node-gather-daemonset-4xsjq\" (UID: \"720358b7-256e-473b-8271-93bb60f8676f\") " pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:45.999331 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:45.999250 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:46.267020 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:46.266974 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq"] Sep 11 00:14:46.268808 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:14:46.268777 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod720358b7_256e_473b_8271_93bb60f8676f.slice/crio-7f60b21650ed1a6518b7c515d975ce4fe682aae805f1ba6b70cff85cf56be6ef WatchSource:0}: Error finding container 7f60b21650ed1a6518b7c515d975ce4fe682aae805f1ba6b70cff85cf56be6ef: Status 404 returned error can't find the container with id 7f60b21650ed1a6518b7c515d975ce4fe682aae805f1ba6b70cff85cf56be6ef Sep 11 00:14:47.272509 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:47.272474 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" event={"ID":"720358b7-256e-473b-8271-93bb60f8676f","Type":"ContainerStarted","Data":"67c2c5310967dc4d7f00f41bc01f969bc71c734bb0770cded58618e47ef9ee6a"} Sep 11 00:14:47.272889 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:47.272525 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" event={"ID":"720358b7-256e-473b-8271-93bb60f8676f","Type":"ContainerStarted","Data":"7f60b21650ed1a6518b7c515d975ce4fe682aae805f1ba6b70cff85cf56be6ef"} Sep 11 00:14:47.272889 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:47.272616 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:47.290646 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:47.290604 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" podStartSLOduration=2.290588816 podStartE2EDuration="2.290588816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:14:47.288451585 +0000 UTC m=+2294.590929597" watchObservedRunningTime="2026-09-11 00:14:47.290588816 +0000 UTC m=+2294.593066828" Sep 11 00:14:47.590224 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:47.590146 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lfxmq_08605f38-1013-4f84-b980-10bada8fa720/dns/0.log" Sep 11 00:14:47.607819 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:47.607795 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lfxmq_08605f38-1013-4f84-b980-10bada8fa720/kube-rbac-proxy/0.log" Sep 11 00:14:47.675315 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:47.675290 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rvpbw_1c0b07bf-513c-4c0f-aed5-bcfd96136254/dns-node-resolver/0.log" Sep 11 00:14:48.324247 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:48.324205 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zlqzj_3750ae5a-25be-4d56-89ae-67210d6cb357/node-ca/0.log" Sep 11 00:14:49.223627 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:49.223575 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn_afc3a1d4-817a-4da5-918e-12c098448c9e/istio-proxy/0.log" Sep 11 00:14:49.809152 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:49.809123 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zcqfs_73466eaa-d499-4209-be94-ba88149422b1/discovery/0.log" Sep 11 00:14:50.864596 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:50.864565 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xlw45_56489c59-0fc9-4469-a76d-141755000565/serve-healthcheck-canary/0.log" Sep 11 00:14:51.328230 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:51.328152 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7s59l_3c94d0ce-50fc-4170-b7ad-87648ed95d4f/kube-rbac-proxy/0.log" Sep 11 00:14:51.348490 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:51.348466 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7s59l_3c94d0ce-50fc-4170-b7ad-87648ed95d4f/exporter/0.log" Sep 11 00:14:51.369988 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:51.369964 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7s59l_3c94d0ce-50fc-4170-b7ad-87648ed95d4f/extractor/0.log" Sep 11 00:14:53.285716 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:53.285688 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-chw9w/perf-node-gather-daemonset-4xsjq" Sep 11 00:14:53.749939 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:53.749912 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-twk4s_7e927681-ac8b-4b2e-b3f8-0fd5c3ac3b2a/server/0.log" Sep 11 00:14:53.930424 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:53.930390 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bb6f54779-5mph5_04334b39-1b36-4759-b81d-877da7cf4e37/manager/0.log" Sep 11 00:14:53.945400 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:53.945373 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bb6f54779-5mph5_04334b39-1b36-4759-b81d-877da7cf4e37/copy-manifests/0.log" Sep 11 00:14:55.212363 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:55.212330 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-98d67b8cf-wswwl_95221866-eb45-44fc-bf0e-0b389769776a/manager/0.log" Sep 11 00:14:55.238857 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:55.238828 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jdnr6_c62f070a-0916-4aba-b33f-d32b9bf0916e/openshift-lws-operator/0.log" Sep 11 00:14:55.743996 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:55.743967 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn_afc3a1d4-817a-4da5-918e-12c098448c9e/istio-proxy/0.log" Sep 11 00:14:56.335855 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:56.335813 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zcqfs_73466eaa-d499-4209-be94-ba88149422b1/discovery/0.log" Sep 11 00:14:58.021788 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:58.021760 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-ghzjn_afc3a1d4-817a-4da5-918e-12c098448c9e/istio-proxy/0.log" Sep 11 00:14:58.621138 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:14:58.621102 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zcqfs_73466eaa-d499-4209-be94-ba88149422b1/discovery/0.log" Sep 11 00:15:00.138473 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:00.138437 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818095-x7vqq"] Sep 11 00:15:00.138871 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:00.138736 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerName="istio-proxy" Sep 11 00:15:00.138871 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:00.138748 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerName="istio-proxy" Sep 11 00:15:00.138871 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:00.138819 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f28b6070-193d-4086-ba4c-b679fd2c84d4" containerName="istio-proxy" Sep 11 00:15:00.144962 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:00.144944 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818095-x7vqq" Sep 11 00:15:00.147513 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:00.147482 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xwtg9\"" Sep 11 00:15:00.150952 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:00.150929 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818095-x7vqq"] Sep 11 00:15:00.155495 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:00.155478 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818095-x7vqq" Sep 11 00:15:00.422682 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:00.422657 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818095-x7vqq"] Sep 11 00:15:00.425867 ip-10-0-134-90 kubenswrapper[2583]: W0911 00:15:00.425835 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09516856_3fa9_4fe7_8289_ab6b3226d5ae.slice/crio-57cc1c7c9864b89a1c28d0058c9bcb76995d8fec351d7ee40097cb1eb3123bcf WatchSource:0}: Error finding container 57cc1c7c9864b89a1c28d0058c9bcb76995d8fec351d7ee40097cb1eb3123bcf: Status 404 returned error can't find the container with id 57cc1c7c9864b89a1c28d0058c9bcb76995d8fec351d7ee40097cb1eb3123bcf Sep 11 00:15:00.438272 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:00.438249 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bvsht_ba502de5-22a2-417b-9c16-e281af5cd43a/migrator/0.log" Sep 11 00:15:00.456930 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:00.456911 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-bvsht_ba502de5-22a2-417b-9c16-e281af5cd43a/graceful-termination/0.log" Sep 11 00:15:01.333256 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:01.333223 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818095-x7vqq" event={"ID":"09516856-3fa9-4fe7-8289-ab6b3226d5ae","Type":"ContainerStarted","Data":"1d83b17212422b803ff32d3a142305cdaca067ed18c71b3104675059511cedcd"} Sep 11 00:15:01.333643 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:01.333279 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818095-x7vqq" event={"ID":"09516856-3fa9-4fe7-8289-ab6b3226d5ae","Type":"ContainerStarted","Data":"57cc1c7c9864b89a1c28d0058c9bcb76995d8fec351d7ee40097cb1eb3123bcf"} Sep 11 00:15:01.351423 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:01.351374 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818095-x7vqq" podStartSLOduration=1.3513579230000001 podStartE2EDuration="1.351357923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:15:01.350215899 +0000 UTC m=+2308.652693911" watchObservedRunningTime="2026-09-11 00:15:01.351357923 +0000 UTC m=+2308.653835935" Sep 11 00:15:01.879575 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:01.879549 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pvxvx_0170eaf4-0b83-4c76-be27-600a18f3226e/kube-multus-additional-cni-plugins/0.log" Sep 11 00:15:01.898342 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:01.898299 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pvxvx_0170eaf4-0b83-4c76-be27-600a18f3226e/egress-router-binary-copy/0.log" Sep 11 00:15:01.922199 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:01.922174 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pvxvx_0170eaf4-0b83-4c76-be27-600a18f3226e/cni-plugins/0.log" Sep 11 00:15:01.940369 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:01.940347 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pvxvx_0170eaf4-0b83-4c76-be27-600a18f3226e/bond-cni-plugin/0.log" Sep 11 00:15:01.963354 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:01.963332 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pvxvx_0170eaf4-0b83-4c76-be27-600a18f3226e/routeoverride-cni/0.log" Sep 11 00:15:01.981591 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:01.981568 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pvxvx_0170eaf4-0b83-4c76-be27-600a18f3226e/whereabouts-cni-bincopy/0.log" Sep 11 00:15:02.001350 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:02.001323 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pvxvx_0170eaf4-0b83-4c76-be27-600a18f3226e/whereabouts-cni/0.log" Sep 11 00:15:02.339981 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:02.339884 2583 generic.go:356] "Generic (PLEG): container finished" podID="09516856-3fa9-4fe7-8289-ab6b3226d5ae" containerID="1d83b17212422b803ff32d3a142305cdaca067ed18c71b3104675059511cedcd" exitCode=0 Sep 11 00:15:02.339981 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:02.339925 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818095-x7vqq" event={"ID":"09516856-3fa9-4fe7-8289-ab6b3226d5ae","Type":"ContainerDied","Data":"1d83b17212422b803ff32d3a142305cdaca067ed18c71b3104675059511cedcd"} Sep 11 00:15:02.360855 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:02.360831 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wnl97_22a3fa41-e1e7-481c-a52a-ffb83f4804cc/kube-multus/0.log" Sep 11 00:15:02.382514 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:02.382489 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7hv8v_e030b330-e967-4317-bcd2-4664364ee362/network-metrics-daemon/0.log" Sep 11 00:15:02.397281 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:02.397255 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7hv8v_e030b330-e967-4317-bcd2-4664364ee362/kube-rbac-proxy/0.log" Sep 11 00:15:03.477863 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:03.477836 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818095-x7vqq" Sep 11 00:15:03.720169 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:03.720134 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rc9rz_0331a4a5-abd1-419c-9116-60b56f90673b/ovn-controller/0.log" Sep 11 00:15:03.735196 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:03.735161 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rc9rz_0331a4a5-abd1-419c-9116-60b56f90673b/ovn-acl-logging/0.log" Sep 11 00:15:03.756452 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:03.756423 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rc9rz_0331a4a5-abd1-419c-9116-60b56f90673b/kube-rbac-proxy-node/0.log" Sep 11 00:15:03.782590 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:03.782558 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rc9rz_0331a4a5-abd1-419c-9116-60b56f90673b/kube-rbac-proxy-ovn-metrics/0.log" Sep 11 00:15:03.799698 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:03.799674 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rc9rz_0331a4a5-abd1-419c-9116-60b56f90673b/northd/0.log" Sep 11 00:15:03.818023 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:03.818000 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rc9rz_0331a4a5-abd1-419c-9116-60b56f90673b/nbdb/0.log" Sep 11 00:15:03.836964 ip-10-0-134-90 kubenswrapper[2583]: I0911 00:15:03.836933 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rc9rz_0331a4a5-abd1-419c-9116-60b56f90673b/sbdb/0.log"