Sep 15 11:36:51.505488 ip-10-0-128-189 systemd[1]: Starting Kubernetes Kubelet... Sep 15 11:36:51.911706 ip-10-0-128-189 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 11:36:51.911706 ip-10-0-128-189 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 15 11:36:51.911706 ip-10-0-128-189 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 11:36:51.911706 ip-10-0-128-189 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 15 11:36:51.911706 ip-10-0-128-189 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 11:36:51.912808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.912723 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 15 11:36:51.916405 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916390 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 11:36:51.916405 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916404 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916408 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916411 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916415 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916418 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916420 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916423 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916426 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916432 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916435 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916437 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916440 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916443 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916445 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916448 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916451 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916453 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916455 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916459 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916463 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 11:36:51.916466 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916467 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916470 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916474 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916477 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916480 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916482 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916485 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916488 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916491 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916493 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916496 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916498 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916501 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916503 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916506 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916508 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916511 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916513 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916518 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916520 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 11:36:51.916937 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916522 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916526 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916528 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916531 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916533 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916536 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916538 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916541 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916543 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916546 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916548 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916551 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916553 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916556 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916559 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916562 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916564 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916567 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916569 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 11:36:51.917434 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916572 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916577 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916580 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916583 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916586 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916588 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916590 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916593 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916595 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916598 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916600 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916604 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916607 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916611 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916613 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916616 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916619 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916623 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916626 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 11:36:51.917958 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916628 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916631 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916634 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916637 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916639 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916642 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916645 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916648 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916651 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916654 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916656 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916659 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916662 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916665 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916667 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916670 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916672 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916675 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916677 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.916680 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 11:36:51.918430 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918216 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918225 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918230 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918233 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918236 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918238 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918241 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918244 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918253 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918256 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918261 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918264 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918267 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918270 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918273 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918276 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918279 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918282 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918284 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 11:36:51.918908 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918288 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918291 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918293 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918296 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918298 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918301 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918303 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918306 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918313 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918316 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918318 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918321 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918324 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918327 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918329 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918332 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918334 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918337 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918339 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918342 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 11:36:51.919386 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918344 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918347 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918349 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918352 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918355 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918357 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918360 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918363 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918366 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918370 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918373 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918376 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918379 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918382 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918386 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918389 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918392 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918395 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918398 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 11:36:51.919870 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918401 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918403 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918406 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918409 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918411 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918415 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918418 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918420 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918423 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918425 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918428 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918431 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918433 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918436 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918439 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918441 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918444 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918447 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918449 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918452 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 11:36:51.920348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918455 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918457 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918460 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918463 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918465 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918468 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918470 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918473 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918475 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918478 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918480 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918483 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918485 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918488 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918490 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918493 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918495 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918497 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918500 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918502 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 11:36:51.920834 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918505 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918853 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918858 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918860 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918863 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918866 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918869 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918872 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918874 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918877 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918880 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918885 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918889 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918891 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918894 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918897 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918899 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918902 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918905 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918908 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 11:36:51.921348 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918910 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918913 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918916 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918918 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918922 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918925 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918928 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918930 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918933 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918935 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918938 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918940 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918943 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918945 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918947 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918950 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918952 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918955 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918957 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 11:36:51.921840 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918960 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918962 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918965 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918967 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918970 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918973 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918976 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918978 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918983 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918986 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918989 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918992 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918994 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.918997 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919000 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919003 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919005 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919008 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919011 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919013 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 15 11:36:51.922318 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919016 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919018 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919022 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919024 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919027 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919029 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919031 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919034 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919036 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919039 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919041 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919043 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919046 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919049 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919051 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919054 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919057 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919060 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919062 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919065 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 11:36:51.922844 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919067 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919070 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919072 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919075 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919077 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919080 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919082 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919085 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919088 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919090 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919093 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919095 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919097 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919100 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919103 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919105 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919108 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919110 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919113 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919115 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 11:36:51.923343 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919118 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919204 2573 flags.go:64] FLAG: --address="0.0.0.0" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919248 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919255 2573 flags.go:64] FLAG: --anonymous-auth="true" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919259 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919264 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919267 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919271 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919275 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919282 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919286 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919289 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919294 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919297 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919300 2573 flags.go:64] FLAG: --cgroup-root="" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919303 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919306 2573 flags.go:64] FLAG: --client-ca-file="" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919309 2573 flags.go:64] FLAG: --cloud-provider="external" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919312 2573 flags.go:64] FLAG: --cluster-dns="[]" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919316 2573 flags.go:64] FLAG: --cluster-domain="" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919319 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919322 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919325 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919329 2573 flags.go:64] FLAG: --container-log-max-files="5" Sep 15 11:36:51.923838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919332 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919335 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919338 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919342 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919345 2573 flags.go:64] FLAG: --contention-profiling="false" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919348 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919351 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919354 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919357 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919361 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919364 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919367 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919369 2573 flags.go:64] FLAG: --enable-load-reader="false" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919372 2573 flags.go:64] FLAG: --enable-server="true" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919375 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919380 2573 flags.go:64] FLAG: --event-burst="100" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919383 2573 flags.go:64] FLAG: --event-qps="50" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919387 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919411 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919432 2573 flags.go:64] FLAG: --eviction-hard="" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919439 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919443 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919448 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919452 2573 flags.go:64] FLAG: --eviction-soft="" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919455 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 15 11:36:51.924423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919459 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919463 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919466 2573 flags.go:64] FLAG: --experimental-mounter-path="" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919470 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919473 2573 flags.go:64] FLAG: --fail-swap-on="true" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919476 2573 flags.go:64] FLAG: --feature-gates="" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919480 2573 flags.go:64] FLAG: --file-check-frequency="20s" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919484 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919487 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919490 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919493 2573 flags.go:64] FLAG: --healthz-port="10248" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919497 2573 flags.go:64] FLAG: --help="false" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919500 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-128-189.ec2.internal" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919504 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919507 2573 flags.go:64] FLAG: --http-check-frequency="20s" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919510 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919514 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919519 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919522 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919525 2573 flags.go:64] FLAG: --image-service-endpoint="" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919528 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919532 2573 flags.go:64] FLAG: --kube-api-burst="100" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919536 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919539 2573 flags.go:64] FLAG: --kube-api-qps="50" Sep 15 11:36:51.925028 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919542 2573 flags.go:64] FLAG: --kube-reserved="" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919545 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919548 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919552 2573 flags.go:64] FLAG: --kubelet-cgroups="" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919555 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919558 2573 flags.go:64] FLAG: --lock-file="" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919561 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919564 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919567 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919573 2573 flags.go:64] FLAG: --log-json-split-stream="false" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919576 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919579 2573 flags.go:64] FLAG: --log-text-split-stream="false" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919582 2573 flags.go:64] FLAG: --logging-format="text" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919585 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919589 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919592 2573 flags.go:64] FLAG: --manifest-url="" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919595 2573 flags.go:64] FLAG: --manifest-url-header="" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919599 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919602 2573 flags.go:64] FLAG: --max-open-files="1000000" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919607 2573 flags.go:64] FLAG: --max-pods="110" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919610 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919613 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919616 2573 flags.go:64] FLAG: --memory-manager-policy="None" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919619 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919623 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 15 11:36:51.925648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919626 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919629 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919636 2573 flags.go:64] FLAG: --node-status-max-images="50" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919640 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919643 2573 flags.go:64] FLAG: --oom-score-adj="-999" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919646 2573 flags.go:64] FLAG: --pod-cidr="" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919649 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919654 2573 flags.go:64] FLAG: --pod-manifest-path="" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919657 2573 flags.go:64] FLAG: --pod-max-pids="-1" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919661 2573 flags.go:64] FLAG: --pods-per-core="0" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919664 2573 flags.go:64] FLAG: --port="10250" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919667 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919671 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0df8f4cd135d03cd4" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919674 2573 flags.go:64] FLAG: --qos-reserved="" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919677 2573 flags.go:64] FLAG: --read-only-port="10255" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919680 2573 flags.go:64] FLAG: --register-node="true" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919682 2573 flags.go:64] FLAG: --register-with-taints="" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919693 2573 flags.go:64] FLAG: --registry-burst="10" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919696 2573 flags.go:64] FLAG: --registry-qps="5" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919698 2573 flags.go:64] FLAG: --reserved-cpus="" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919701 2573 flags.go:64] FLAG: --reserved-memory="" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919705 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919708 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919711 2573 flags.go:64] FLAG: --rotate-certificates="false" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919714 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 15 11:36:51.926313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919717 2573 flags.go:64] FLAG: --runonce="false" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919720 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919723 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919726 2573 flags.go:64] FLAG: --seccomp-default="false" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919729 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919732 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919735 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919738 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919742 2573 flags.go:64] FLAG: --storage-driver-password="root" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919745 2573 flags.go:64] FLAG: --storage-driver-secure="false" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919748 2573 flags.go:64] FLAG: --storage-driver-table="stats" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919751 2573 flags.go:64] FLAG: --storage-driver-user="root" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919754 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919758 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919761 2573 flags.go:64] FLAG: --system-cgroups="" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919765 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919771 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919774 2573 flags.go:64] FLAG: --tls-cert-file="" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919777 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919805 2573 flags.go:64] FLAG: --tls-min-version="" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919809 2573 flags.go:64] FLAG: --tls-private-key-file="" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919812 2573 flags.go:64] FLAG: --topology-manager-policy="none" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919816 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919819 2573 flags.go:64] FLAG: --topology-manager-scope="container" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919822 2573 flags.go:64] FLAG: --v="2" Sep 15 11:36:51.926913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919826 2573 flags.go:64] FLAG: --version="false" Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919830 2573 flags.go:64] FLAG: --vmodule="" Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919834 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.919837 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919963 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919966 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919970 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919973 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919976 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919982 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919984 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919987 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919990 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919993 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919995 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.919998 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920000 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920003 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920006 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920008 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 11:36:51.927563 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920011 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920014 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920017 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920022 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920024 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920027 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920030 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920032 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920035 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920037 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920040 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920042 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920045 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920047 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920050 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920053 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920055 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920058 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920061 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920064 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 11:36:51.928056 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920066 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920070 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920073 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920076 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920078 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920081 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920083 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920086 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920089 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920091 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920094 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920096 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920099 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920102 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920105 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920109 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920112 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920114 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920117 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920120 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 11:36:51.928640 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920122 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920125 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920127 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920130 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920133 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920135 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920138 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920142 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920145 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920149 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920152 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920155 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920157 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920161 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920163 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920166 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920168 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920171 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920174 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 11:36:51.929129 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920176 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920180 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920198 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920201 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920203 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920206 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920209 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920211 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920217 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920221 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920224 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920226 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920229 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920232 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920234 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920237 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920240 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920242 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 11:36:51.929650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920245 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 11:36:51.930102 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920247 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 11:36:51.930102 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920250 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 11:36:51.930102 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920253 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 11:36:51.930102 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920255 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 11:36:51.930102 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.920258 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 11:36:51.930102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.921169 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 11:36:51.930102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.929079 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 15 11:36:51.930102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.929097 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 15 11:36:51.930102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.930035 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 15 11:36:51.930102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.930046 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930122 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930128 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930131 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930134 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930137 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930140 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930143 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930145 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930148 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930151 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930154 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930156 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930159 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930161 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930164 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930167 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930170 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930172 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930175 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 11:36:51.930425 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930177 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930180 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930183 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930199 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930202 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930204 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930207 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930209 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930212 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930221 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930224 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930226 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930229 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930231 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930235 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930239 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930243 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930246 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930249 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 11:36:51.930930 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930252 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930255 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930258 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930262 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930266 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930268 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930271 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930274 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930276 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930279 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930281 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930284 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930287 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930289 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930292 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930294 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930297 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930299 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930302 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 11:36:51.931403 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930304 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930307 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930309 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930312 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930320 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930322 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930325 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930327 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930330 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930333 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930336 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930338 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930341 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930343 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930346 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930349 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930351 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930354 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930356 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 11:36:51.931855 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930359 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930361 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930364 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930366 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930368 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930371 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930373 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930376 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930378 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930381 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930383 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930386 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930390 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930394 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930397 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930405 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930409 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930412 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930421 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 11:36:51.932505 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930424 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930427 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930430 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930433 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.930438 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930587 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930591 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930594 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930597 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930600 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930603 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930606 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930609 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930612 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930615 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 11:36:51.933209 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930618 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930620 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930623 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930626 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930629 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930631 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930634 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930637 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930639 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930642 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930645 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930648 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930651 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930654 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930656 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930659 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930662 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930669 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930672 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930675 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 11:36:51.933682 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930677 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930680 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930683 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930685 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930687 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930690 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930693 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930695 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930697 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930700 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930703 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930705 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930709 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930712 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930715 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930718 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930721 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930724 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930726 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 11:36:51.934461 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930729 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930732 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930734 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930737 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930740 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930742 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930744 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930747 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930749 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930752 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930755 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930762 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930765 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930768 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930771 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930774 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930776 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930778 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930781 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930783 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 11:36:51.934939 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930786 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930788 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930791 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930793 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930796 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930798 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930801 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930803 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930806 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930808 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930811 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930813 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930816 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930819 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930822 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930825 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930827 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930830 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930832 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 11:36:51.935629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930835 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 11:36:51.936117 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930837 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 11:36:51.936117 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930840 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 11:36:51.936117 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930842 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 11:36:51.936117 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930845 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 11:36:51.936117 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930853 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 11:36:51.936117 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930857 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 11:36:51.936117 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930860 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 11:36:51.936117 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930862 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 11:36:51.936117 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930865 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 11:36:51.936117 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:51.930867 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 11:36:51.936117 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.930872 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 11:36:51.936117 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.930970 2573 server.go:962] "Client rotation is on, will bootstrap in background" Sep 15 11:36:51.936117 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.935089 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 15 11:36:51.936505 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.936223 2573 server.go:1019] "Starting client certificate rotation" Sep 15 11:36:51.936505 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.936325 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 11:36:51.936505 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.936356 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 11:36:51.939233 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.939172 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 11:36:51.940341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.940324 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 11:36:51.953496 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.953479 2573 log.go:25] "Validated CRI v1 runtime API" Sep 15 11:36:51.957384 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.957366 2573 log.go:25] "Validated CRI v1 image API" Sep 15 11:36:51.958318 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.958302 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 15 11:36:51.961215 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.961177 2573 fs.go:135] Filesystem UUIDs: map[085e8bf3-98e7-4a2a-a71e-c0f09ce723ba:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f60fb978-44e4-430b-8061-7b47d196dfc0:/dev/nvme0n1p4] Sep 15 11:36:51.961271 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.961215 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 15 11:36:51.967613 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.967502 2573 manager.go:216] Machine: {Timestamp:2026-09-15 11:36:51.965109593 +0000 UTC m=+0.355871564 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099637 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bd9224ce3aeb3e0b47aa17d2890c9 SystemUUID:ec2bd922-4ce3-aeb3-e0b4-7aa17d2890c9 BootID:73b66d79-4246-4f4f-8f13-fae5a90e88e8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d2:60:7d:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d2:60:7d:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:cd:69:b3:07:32 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 15 11:36:51.967613 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.967607 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 15 11:36:51.967729 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.967718 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 15 11:36:51.970589 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.970559 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 15 11:36:51.970729 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.970592 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-189.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 15 11:36:51.970780 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.970767 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 11:36:51.971434 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.971423 2573 topology_manager.go:138] "Creating topology manager with none policy" Sep 15 11:36:51.971468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.971436 2573 container_manager_linux.go:309] "Creating device plugin manager" Sep 15 11:36:51.971468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.971448 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 11:36:51.972064 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.972055 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 11:36:51.973242 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.973232 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 15 11:36:51.976709 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.976697 2573 state_mem.go:36] "Initialized new in-memory state store" Sep 15 11:36:51.976827 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.976818 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 15 11:36:51.976858 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.976848 2573 kubelet.go:486] "Attempting to sync node with API server" Sep 15 11:36:51.976885 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.976861 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 15 11:36:51.976913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.976885 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 15 11:36:51.976913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.976899 2573 kubelet.go:398] "Adding apiserver pod source" Sep 15 11:36:51.976913 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.976912 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 15 11:36:51.977323 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.977311 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 11:36:51.977362 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.977337 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 11:36:51.979765 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.979743 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 15 11:36:51.983523 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.983505 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 15 11:36:51.983604 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.983546 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 15 11:36:51.985532 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.985518 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 15 11:36:51.985616 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.985539 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 15 11:36:51.985616 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.985549 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 15 11:36:51.985616 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.985557 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 15 11:36:51.985616 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.985569 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 15 11:36:51.985616 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.985579 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 15 11:36:51.985616 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.985599 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 15 11:36:51.985616 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.985608 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 15 11:36:51.985840 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.985619 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 15 11:36:51.985840 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.985629 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 15 11:36:51.985840 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.985643 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 15 11:36:51.985840 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.985652 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 15 11:36:51.987477 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.987465 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 15 11:36:51.987532 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.987487 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 15 11:36:51.989466 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:51.989440 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 15 11:36:51.989466 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:51.989441 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-189.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 15 11:36:51.991434 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.991418 2573 server.go:1268] "Started kubelet" Sep 15 11:36:51.991517 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.991485 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 15 11:36:51.991566 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.991494 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 15 11:36:51.991566 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.991543 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 15 11:36:51.992100 ip-10-0-128-189 systemd[1]: Started Kubernetes Kubelet. Sep 15 11:36:51.992948 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.992932 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 15 11:36:51.993594 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.993580 2573 server.go:310] "Adding debug handlers to kubelet server" Sep 15 11:36:51.997259 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:51.997235 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 15 11:36:51.999952 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:51.999935 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-189.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 15 11:36:52.000047 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.000029 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 15 11:36:52.000773 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.000750 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 15 11:36:52.000884 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.000851 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:52.001230 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:51.999961 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-189.ec2.internal.18d57b7eeb2dea03 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-189.ec2.internal,UID:ip-10-0-128-189.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-189.ec2.internal,},FirstTimestamp:2026-09-15 11:36:51.991390723 +0000 UTC m=+0.382152694,LastTimestamp:2026-09-15 11:36:51.991390723 +0000 UTC m=+0.382152694,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-189.ec2.internal,}" Sep 15 11:36:52.001483 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.001467 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 15 11:36:52.001483 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.001468 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 15 11:36:52.001618 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.001497 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 15 11:36:52.001618 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.001552 2573 reconstruct.go:97] "Volume reconstruction finished" Sep 15 11:36:52.001618 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.001563 2573 reconciler.go:29] "Reconciler: start to sync state" Sep 15 11:36:52.001793 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.001778 2573 factory.go:153] Registering CRI-O factory Sep 15 11:36:52.001793 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.001795 2573 factory.go:223] Registration of the crio container factory successfully Sep 15 11:36:52.001920 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.001907 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 15 11:36:52.001982 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.001920 2573 factory.go:55] Registering systemd factory Sep 15 11:36:52.001982 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.001930 2573 factory.go:223] Registration of the systemd container factory successfully Sep 15 11:36:52.001982 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.001946 2573 factory.go:103] Registering Raw factory Sep 15 11:36:52.001982 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.001957 2573 manager.go:1196] Started watching for new ooms in manager Sep 15 11:36:52.002374 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.002361 2573 manager.go:349] Starting recovery of all containers Sep 15 11:36:52.006071 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.006029 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-189.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 15 11:36:52.006223 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.006145 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 15 11:36:52.012147 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.012119 2573 manager.go:354] Recovery completed Sep 15 11:36:52.017915 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.017890 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-76n86" Sep 15 11:36:52.019132 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.019120 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:36:52.021068 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.021048 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:36:52.021141 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.021083 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:36:52.021141 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.021098 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:36:52.021700 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.021687 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 15 11:36:52.021700 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.021698 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 15 11:36:52.021797 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.021711 2573 state_mem.go:36] "Initialized new in-memory state store" Sep 15 11:36:52.021797 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.021731 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:36:52.022556 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.022536 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:36:52.022634 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.022572 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:36:52.022634 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.022588 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:36:52.022926 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.022908 2573 policy_none.go:49] "None policy: Start" Sep 15 11:36:52.022926 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.022929 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 15 11:36:52.023052 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.022942 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 15 11:36:52.023052 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.022971 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-189.ec2.internal.18d57b7eecf2be10 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-189.ec2.internal,UID:ip-10-0-128-189.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-189.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-189.ec2.internal,},FirstTimestamp:2026-09-15 11:36:52.02106728 +0000 UTC m=+0.411829252,LastTimestamp:2026-09-15 11:36:52.02106728 +0000 UTC m=+0.411829252,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-189.ec2.internal,}" Sep 15 11:36:52.023919 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.023903 2573 policy_none.go:47] "Start" Sep 15 11:36:52.024740 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.024724 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-76n86" Sep 15 11:36:52.050785 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.050767 2573 manager.go:337] "Starting Device Plugin manager" Sep 15 11:36:52.066387 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.050823 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 15 11:36:52.066387 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.050843 2573 server.go:87] "Starting device plugin registration server" Sep 15 11:36:52.066387 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.051158 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 15 11:36:52.066387 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.051175 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 15 11:36:52.066387 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.051366 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 15 11:36:52.066387 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.051435 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 15 11:36:52.066387 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.051443 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 15 11:36:52.066387 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.052524 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 15 11:36:52.066387 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.052572 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:52.066387 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.066123 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 15 11:36:52.067235 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.067204 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 15 11:36:52.067280 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.067247 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 15 11:36:52.067280 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.067268 2573 kubelet.go:2448] "Starting kubelet main sync loop" Sep 15 11:36:52.067364 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.067301 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 15 11:36:52.069534 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.069516 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 11:36:52.151542 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.151512 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:36:52.152459 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.152443 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:36:52.152574 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.152470 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:36:52.152574 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.152481 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:36:52.152574 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.152501 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.161938 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.161875 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.161938 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.161896 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-189.ec2.internal\": node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:52.168403 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.168377 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-189.ec2.internal"] Sep 15 11:36:52.168491 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.168440 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:36:52.169236 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.169220 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:36:52.169301 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.169248 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:36:52.169301 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.169257 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:36:52.170472 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.170460 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:36:52.170613 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.170597 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.170657 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.170628 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:36:52.171130 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.171117 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:36:52.171175 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.171141 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:36:52.171175 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.171150 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:36:52.171259 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.171176 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:36:52.171259 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.171221 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:36:52.171259 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.171233 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:36:52.172281 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.172263 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.172357 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.172285 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:36:52.172920 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.172907 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:36:52.172978 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.172934 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:36:52.172978 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.172944 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:36:52.186146 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.186130 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:52.202976 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.202956 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f8e4755d9e46fb80d2e9e4a381fbae35-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal\" (UID: \"f8e4755d9e46fb80d2e9e4a381fbae35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.203049 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.202979 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8e4755d9e46fb80d2e9e4a381fbae35-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal\" (UID: \"f8e4755d9e46fb80d2e9e4a381fbae35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.203049 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.202999 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0cce461e99c8c476d4a7a451016bffc0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-189.ec2.internal\" (UID: \"0cce461e99c8c476d4a7a451016bffc0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.209903 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.209888 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-189.ec2.internal\" not found" node="ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.214249 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.214235 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-189.ec2.internal\" not found" node="ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.287027 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.287007 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:52.303342 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.303322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f8e4755d9e46fb80d2e9e4a381fbae35-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal\" (UID: \"f8e4755d9e46fb80d2e9e4a381fbae35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.303407 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.303351 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8e4755d9e46fb80d2e9e4a381fbae35-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal\" (UID: \"f8e4755d9e46fb80d2e9e4a381fbae35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.303407 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.303370 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0cce461e99c8c476d4a7a451016bffc0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-189.ec2.internal\" (UID: \"0cce461e99c8c476d4a7a451016bffc0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.303479 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.303421 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f8e4755d9e46fb80d2e9e4a381fbae35-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal\" (UID: \"f8e4755d9e46fb80d2e9e4a381fbae35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.303479 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.303421 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0cce461e99c8c476d4a7a451016bffc0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-189.ec2.internal\" (UID: \"0cce461e99c8c476d4a7a451016bffc0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.303479 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.303446 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8e4755d9e46fb80d2e9e4a381fbae35-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal\" (UID: \"f8e4755d9e46fb80d2e9e4a381fbae35\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.387501 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.387476 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:52.488283 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.488216 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:52.513696 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.513681 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.518845 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.518538 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-189.ec2.internal" Sep 15 11:36:52.588861 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.588838 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:52.689329 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.689297 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:52.789896 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.789836 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:52.864870 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.864854 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 11:36:52.890112 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:52.890092 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:52.935612 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.935587 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 15 11:36:52.936002 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.935706 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 11:36:52.936002 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:52.935736 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 11:36:54.151882 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:54.151858 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:54.192087 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:54.192039 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 11:31:53 +0000 UTC" deadline="2028-05-25 20:24:52.436299146 +0000 UTC" Sep 15 11:36:54.192208 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:54.192112 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14840h47m58.244197146s" Sep 15 11:36:54.252511 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:54.252486 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:54.263247 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:54.263223 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 11:36:54.316406 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:54.316374 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cce461e99c8c476d4a7a451016bffc0.slice/crio-ad8c33f7f9974cdd285c2801ec0e8119caba8a5fbc029406d4d07775a011ca07 WatchSource:0}: Error finding container ad8c33f7f9974cdd285c2801ec0e8119caba8a5fbc029406d4d07775a011ca07: Status 404 returned error can't find the container with id ad8c33f7f9974cdd285c2801ec0e8119caba8a5fbc029406d4d07775a011ca07 Sep 15 11:36:54.316593 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:54.316574 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e4755d9e46fb80d2e9e4a381fbae35.slice/crio-e2928e4cb0cee0ec8c62f8313421c8d10386abc919cf25a61aaee860f6385106 WatchSource:0}: Error finding container e2928e4cb0cee0ec8c62f8313421c8d10386abc919cf25a61aaee860f6385106: Status 404 returned error can't find the container with id e2928e4cb0cee0ec8c62f8313421c8d10386abc919cf25a61aaee860f6385106 Sep 15 11:36:54.320721 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:54.320705 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 11:36:54.352574 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:54.352546 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:54.453080 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:54.453056 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:54.553628 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:54.553594 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-189.ec2.internal\" not found" Sep 15 11:36:54.585176 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:54.585142 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 11:36:54.662678 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:54.662652 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal" Sep 15 11:36:54.675697 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:54.675679 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 11:36:54.676862 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:54.676849 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-189.ec2.internal" Sep 15 11:36:54.683474 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:54.683458 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 11:36:55.138785 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.138754 2573 apiserver.go:52] "Watching apiserver" Sep 15 11:36:55.147381 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.147349 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 15 11:36:55.148259 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.148232 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-ldmmr","openshift-network-diagnostics/network-check-target-zdv99","kube-system/konnectivity-agent-xqnpw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9","openshift-cluster-node-tuning-operator/tuned-tdfp5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal","openshift-network-operator/iptables-alerter-pmkbv","openshift-ovn-kubernetes/ovnkube-node-kj4f2","kube-system/global-pull-secret-syncer-msl5t","kube-system/kube-apiserver-proxy-ip-10-0-128-189.ec2.internal","openshift-dns/node-resolver-z8grp","openshift-image-registry/node-ca-2sll8","openshift-multus/multus-additional-cni-plugins-m4mzd","openshift-multus/multus-c9qbq"] Sep 15 11:36:55.150455 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.150281 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pmkbv" Sep 15 11:36:55.152460 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.152437 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.153249 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.153522 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.153779 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.154049 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xqnpw" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.154058 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tjsrz\"" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.155168 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.155371 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.155561 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.155768 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.155991 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.156211 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-trg4l\"" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.156694 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.156971 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.157468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.156971 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.158137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.157600 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 15 11:36:55.158137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.157693 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2dlgm\"" Sep 15 11:36:55.158575 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.158555 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.159615 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.159595 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nmnlp\"" Sep 15 11:36:55.159825 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.159791 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 15 11:36:55.160228 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.160086 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:55.160228 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.160143 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:36:55.161698 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.161114 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 15 11:36:55.161698 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.161257 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 15 11:36:55.161698 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.161515 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 15 11:36:55.161698 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.161576 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 15 11:36:55.161889 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.161715 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:36:55.161889 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.161793 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:36:55.161961 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.161896 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 15 11:36:55.162348 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.161986 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 15 11:36:55.162348 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.162000 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cwj6n\"" Sep 15 11:36:55.162348 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.162204 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sqfw4\"" Sep 15 11:36:55.163527 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.163257 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:36:55.163527 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.163347 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:36:55.166046 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.166016 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.167897 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.167867 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z8grp" Sep 15 11:36:55.168476 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.168451 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 15 11:36:55.168556 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.168451 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 15 11:36:55.168983 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.168965 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6vm9r\"" Sep 15 11:36:55.169106 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.168992 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 15 11:36:55.169106 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.169091 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 15 11:36:55.169219 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.168965 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 15 11:36:55.169272 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.169227 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 15 11:36:55.169323 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.169307 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2sll8" Sep 15 11:36:55.170345 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.170206 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 15 11:36:55.170345 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.170338 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p9zkp\"" Sep 15 11:36:55.170476 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.170360 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 15 11:36:55.171952 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.171932 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-582qp\"" Sep 15 11:36:55.172132 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.171963 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 15 11:36:55.172309 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.171993 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 15 11:36:55.172543 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.172526 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 15 11:36:55.174920 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.174898 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-sys\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.175050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.174933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-var-lib-kubelet\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.175050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.174959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-sysctl-conf\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.175050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.174983 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-host\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.175050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175001 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-run-k8s-cni-cncf-io\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.175050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:55.175309 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175056 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cef09afa-b154-4fb4-b2cb-bcc07a390e40-iptables-alerter-script\") pod \"iptables-alerter-pmkbv\" (UID: \"cef09afa-b154-4fb4-b2cb-bcc07a390e40\") " pod="openshift-network-operator/iptables-alerter-pmkbv" Sep 15 11:36:55.175309 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175081 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.175309 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175108 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-tmp\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.175309 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175125 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-dbus\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:55.175309 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175141 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-sysconfig\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.175309 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175155 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-kubelet-config\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:55.175309 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6w4\" (UniqueName: \"kubernetes.io/projected/91f95d2b-d44c-4f8a-83bf-649275a608e7-kube-api-access-8m6w4\") pod \"network-metrics-daemon-ldmmr\" (UID: \"91f95d2b-d44c-4f8a-83bf-649275a608e7\") " pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:36:55.175309 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175211 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqbk\" (UniqueName: \"kubernetes.io/projected/4b6f665e-7431-4599-b9ef-302eb545b1c9-kube-api-access-khqbk\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.175309 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175261 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-multus-cni-dir\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.175309 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175300 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-hostroot\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175334 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvq4t\" (UniqueName: \"kubernetes.io/projected/9935a605-d4d3-4471-9740-cb993a66fe98-kube-api-access-vvq4t\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175372 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b6f665e-7431-4599-b9ef-302eb545b1c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175413 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b6f665e-7431-4599-b9ef-302eb545b1c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175444 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-multus-conf-dir\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b6f665e-7431-4599-b9ef-302eb545b1c9-system-cni-dir\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175482 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k92tx\" (UniqueName: \"kubernetes.io/projected/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-kube-api-access-k92tx\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-run-multus-certs\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175567 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-etc-kubernetes\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175600 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7787af06-5984-4095-ac98-a920406f8b40-agent-certs\") pod \"konnectivity-agent-xqnpw\" (UID: \"7787af06-5984-4095-ac98-a920406f8b40\") " pod="kube-system/konnectivity-agent-xqnpw" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175635 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175660 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-var-lib-cni-multus\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175687 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b6f665e-7431-4599-b9ef-302eb545b1c9-cnibin\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b6f665e-7431-4599-b9ef-302eb545b1c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-device-dir\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.175822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9935a605-d4d3-4471-9740-cb993a66fe98-multus-daemon-config\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175831 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jxf9\" (UniqueName: \"kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9\") pod \"network-check-target-zdv99\" (UID: \"ffa5fd6d-96fc-4cd4-ad93-269a473cd51b\") " pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175863 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-registration-dir\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175882 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9935a605-d4d3-4471-9740-cb993a66fe98-cni-binary-copy\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175895 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-kubernetes\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175907 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-var-lib-kubelet\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175925 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs\") pod \"network-metrics-daemon-ldmmr\" (UID: \"91f95d2b-d44c-4f8a-83bf-649275a608e7\") " pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-socket-dir\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.175982 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-system-cni-dir\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176016 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-multus-socket-dir-parent\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cef09afa-b154-4fb4-b2cb-bcc07a390e40-host-slash\") pod \"iptables-alerter-pmkbv\" (UID: \"cef09afa-b154-4fb4-b2cb-bcc07a390e40\") " pod="openshift-network-operator/iptables-alerter-pmkbv" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8229p\" (UniqueName: \"kubernetes.io/projected/cef09afa-b154-4fb4-b2cb-bcc07a390e40-kube-api-access-8229p\") pod \"iptables-alerter-pmkbv\" (UID: \"cef09afa-b154-4fb4-b2cb-bcc07a390e40\") " pod="openshift-network-operator/iptables-alerter-pmkbv" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176118 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-sys-fs\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-modprobe-d\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176163 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-systemd\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-cnibin\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.176499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176207 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-run-netns\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.177250 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176238 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-var-lib-cni-bin\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.177250 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176263 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-run\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.177250 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7787af06-5984-4095-ac98-a920406f8b40-konnectivity-ca\") pod \"konnectivity-agent-xqnpw\" (UID: \"7787af06-5984-4095-ac98-a920406f8b40\") " pod="kube-system/konnectivity-agent-xqnpw" Sep 15 11:36:55.177250 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-sysctl-d\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.177250 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176326 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-tuned\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.177250 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176351 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fkjd\" (UniqueName: \"kubernetes.io/projected/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-kube-api-access-8fkjd\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.177250 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-lib-modules\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.177250 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.176403 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b6f665e-7431-4599-b9ef-302eb545b1c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.181499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.181477 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 11:36:55.195369 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.195345 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vd69z" Sep 15 11:36:55.205929 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.205905 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vd69z" Sep 15 11:36:55.233674 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.233635 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-189.ec2.internal" event={"ID":"0cce461e99c8c476d4a7a451016bffc0","Type":"ContainerStarted","Data":"ad8c33f7f9974cdd285c2801ec0e8119caba8a5fbc029406d4d07775a011ca07"} Sep 15 11:36:55.235295 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.235274 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal" event={"ID":"f8e4755d9e46fb80d2e9e4a381fbae35","Type":"ContainerStarted","Data":"e2928e4cb0cee0ec8c62f8313421c8d10386abc919cf25a61aaee860f6385106"} Sep 15 11:36:55.264194 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.264000 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 15 11:36:55.276961 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.276936 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-hostroot\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.277102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.276974 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvq4t\" (UniqueName: \"kubernetes.io/projected/9935a605-d4d3-4471-9740-cb993a66fe98-kube-api-access-vvq4t\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.277102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b6f665e-7431-4599-b9ef-302eb545b1c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.277102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b6f665e-7431-4599-b9ef-302eb545b1c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.277102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277077 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/087f48fe-aefc-4e62-b27f-d690c27cb027-ovnkube-script-lib\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.277102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277077 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-hostroot\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.277347 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cccg\" (UniqueName: \"kubernetes.io/projected/087f48fe-aefc-4e62-b27f-d690c27cb027-kube-api-access-9cccg\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.277347 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277166 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srs6s\" (UniqueName: \"kubernetes.io/projected/14fd92d1-ab33-45f3-b40f-10048dda0c33-kube-api-access-srs6s\") pod \"node-ca-2sll8\" (UID: \"14fd92d1-ab33-45f3-b40f-10048dda0c33\") " pod="openshift-image-registry/node-ca-2sll8" Sep 15 11:36:55.277347 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277197 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-multus-conf-dir\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.277347 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277223 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b6f665e-7431-4599-b9ef-302eb545b1c9-system-cni-dir\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.277347 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277250 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-run-openvswitch\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.277347 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277279 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/087f48fe-aefc-4e62-b27f-d690c27cb027-ovn-node-metrics-cert\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.277347 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4cec296-04c2-48ae-8024-09e47e456b59-tmp-dir\") pod \"node-resolver-z8grp\" (UID: \"b4cec296-04c2-48ae-8024-09e47e456b59\") " pod="openshift-dns/node-resolver-z8grp" Sep 15 11:36:55.277347 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277323 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b6f665e-7431-4599-b9ef-302eb545b1c9-system-cni-dir\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.277347 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277329 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k92tx\" (UniqueName: \"kubernetes.io/projected/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-kube-api-access-k92tx\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.277625 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277378 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-run-multus-certs\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.277625 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-etc-kubernetes\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.277625 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7787af06-5984-4095-ac98-a920406f8b40-agent-certs\") pod \"konnectivity-agent-xqnpw\" (UID: \"7787af06-5984-4095-ac98-a920406f8b40\") " pod="kube-system/konnectivity-agent-xqnpw" Sep 15 11:36:55.277625 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.277625 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277478 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-var-lib-cni-multus\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.277625 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277504 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b6f665e-7431-4599-b9ef-302eb545b1c9-cnibin\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.277625 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277530 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b6f665e-7431-4599-b9ef-302eb545b1c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.277625 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-device-dir\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.277857 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277626 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-multus-conf-dir\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.277857 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b6f665e-7431-4599-b9ef-302eb545b1c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.277857 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277667 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-slash\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.277857 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277697 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-run-netns\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.277857 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277704 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b6f665e-7431-4599-b9ef-302eb545b1c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.277857 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277728 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9935a605-d4d3-4471-9740-cb993a66fe98-multus-daemon-config\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.277857 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277747 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-var-lib-cni-multus\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.277857 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxf9\" (UniqueName: \"kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9\") pod \"network-check-target-zdv99\" (UID: \"ffa5fd6d-96fc-4cd4-ad93-269a473cd51b\") " pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:36:55.277857 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277788 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-device-dir\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.277857 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277788 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.277857 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277832 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b6f665e-7431-4599-b9ef-302eb545b1c9-cnibin\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.277857 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-registration-dir\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-systemd-units\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277866 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-run-multus-certs\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277899 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9935a605-d4d3-4471-9740-cb993a66fe98-cni-binary-copy\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277915 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-registration-dir\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277949 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277964 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.277975 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b4cec296-04c2-48ae-8024-09e47e456b59-hosts-file\") pod \"node-resolver-z8grp\" (UID: \"b4cec296-04c2-48ae-8024-09e47e456b59\") " pod="openshift-dns/node-resolver-z8grp" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-kubernetes\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-var-lib-kubelet\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278096 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs\") pod \"network-metrics-daemon-ldmmr\" (UID: \"91f95d2b-d44c-4f8a-83bf-649275a608e7\") " pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278122 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-socket-dir\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278185 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-system-cni-dir\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-multus-socket-dir-parent\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cef09afa-b154-4fb4-b2cb-bcc07a390e40-host-slash\") pod \"iptables-alerter-pmkbv\" (UID: \"cef09afa-b154-4fb4-b2cb-bcc07a390e40\") " pod="openshift-network-operator/iptables-alerter-pmkbv" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278329 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8229p\" (UniqueName: \"kubernetes.io/projected/cef09afa-b154-4fb4-b2cb-bcc07a390e40-kube-api-access-8229p\") pod \"iptables-alerter-pmkbv\" (UID: \"cef09afa-b154-4fb4-b2cb-bcc07a390e40\") " pod="openshift-network-operator/iptables-alerter-pmkbv" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278388 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-var-lib-kubelet\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.278413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278395 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-sys-fs\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278433 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9935a605-d4d3-4471-9740-cb993a66fe98-multus-daemon-config\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278434 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/087f48fe-aefc-4e62-b27f-d690c27cb027-env-overrides\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.278464 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14fd92d1-ab33-45f3-b40f-10048dda0c33-serviceca\") pod \"node-ca-2sll8\" (UID: \"14fd92d1-ab33-45f3-b40f-10048dda0c33\") " pod="openshift-image-registry/node-ca-2sll8" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278509 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-etc-kubernetes\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278538 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-multus-socket-dir-parent\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278553 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-kubernetes\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278554 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9935a605-d4d3-4471-9740-cb993a66fe98-cni-binary-copy\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.278584 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs podName:91f95d2b-d44c-4f8a-83bf-649275a608e7 nodeName:}" failed. No retries permitted until 2026-09-15 11:36:55.778548358 +0000 UTC m=+3.008474891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs") pod "network-metrics-daemon-ldmmr" (UID: "91f95d2b-d44c-4f8a-83bf-649275a608e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278590 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-sys-fs\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278601 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cef09afa-b154-4fb4-b2cb-bcc07a390e40-host-slash\") pod \"iptables-alerter-pmkbv\" (UID: \"cef09afa-b154-4fb4-b2cb-bcc07a390e40\") " pod="openshift-network-operator/iptables-alerter-pmkbv" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278603 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-system-cni-dir\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-modprobe-d\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278695 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-socket-dir\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278777 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-systemd\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278805 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b6f665e-7431-4599-b9ef-302eb545b1c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.278950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278820 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-cnibin\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278868 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-modprobe-d\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278876 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-systemd\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-run-netns\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278934 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-cnibin\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.278980 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-run-netns\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-var-lib-cni-bin\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279070 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-var-lib-cni-bin\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279098 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-kubelet\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-var-lib-openvswitch\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279163 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-log-socket\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279182 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-run\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279205 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7787af06-5984-4095-ac98-a920406f8b40-konnectivity-ca\") pod \"konnectivity-agent-xqnpw\" (UID: \"7787af06-5984-4095-ac98-a920406f8b40\") " pod="kube-system/konnectivity-agent-xqnpw" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-run-ovn-kubernetes\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-cni-bin\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279254 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-run\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279267 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-cni-netd\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.279663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/087f48fe-aefc-4e62-b27f-d690c27cb027-ovnkube-config\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279317 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkfxt\" (UniqueName: \"kubernetes.io/projected/b4cec296-04c2-48ae-8024-09e47e456b59-kube-api-access-fkfxt\") pod \"node-resolver-z8grp\" (UID: \"b4cec296-04c2-48ae-8024-09e47e456b59\") " pod="openshift-dns/node-resolver-z8grp" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279341 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-sysctl-d\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-tuned\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fkjd\" (UniqueName: \"kubernetes.io/projected/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-kube-api-access-8fkjd\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-lib-modules\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b6f665e-7431-4599-b9ef-302eb545b1c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279452 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-sysctl-d\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279458 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14fd92d1-ab33-45f3-b40f-10048dda0c33-host\") pod \"node-ca-2sll8\" (UID: \"14fd92d1-ab33-45f3-b40f-10048dda0c33\") " pod="openshift-image-registry/node-ca-2sll8" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-sys\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279533 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-var-lib-kubelet\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279556 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-run-ovn\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-sysctl-conf\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279603 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-host\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279629 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-run-k8s-cni-cncf-io\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279654 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cef09afa-b154-4fb4-b2cb-bcc07a390e40-iptables-alerter-script\") pod \"iptables-alerter-pmkbv\" (UID: \"cef09afa-b154-4fb4-b2cb-bcc07a390e40\") " pod="openshift-network-operator/iptables-alerter-pmkbv" Sep 15 11:36:55.280341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279705 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7787af06-5984-4095-ac98-a920406f8b40-konnectivity-ca\") pod \"konnectivity-agent-xqnpw\" (UID: \"7787af06-5984-4095-ac98-a920406f8b40\") " pod="kube-system/konnectivity-agent-xqnpw" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279741 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-node-log\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279785 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-tmp\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279813 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-dbus\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279837 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-run-systemd\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279846 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-sysctl-conf\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-sysconfig\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279914 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-sysconfig\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279919 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-host\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-kubelet-config\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279971 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6w4\" (UniqueName: \"kubernetes.io/projected/91f95d2b-d44c-4f8a-83bf-649275a608e7-kube-api-access-8m6w4\") pod \"network-metrics-daemon-ldmmr\" (UID: \"91f95d2b-d44c-4f8a-83bf-649275a608e7\") " pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.279975 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.279998 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqbk\" (UniqueName: \"kubernetes.io/projected/4b6f665e-7431-4599-b9ef-302eb545b1c9-kube-api-access-khqbk\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.280045 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-etc-openvswitch\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.280057 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret podName:ecdd2ee0-6fba-4398-ad1b-87c1a4dca820 nodeName:}" failed. No retries permitted until 2026-09-15 11:36:55.780040035 +0000 UTC m=+3.009966553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret") pod "global-pull-secret-syncer-msl5t" (UID: "ecdd2ee0-6fba-4398-ad1b-87c1a4dca820") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.280090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-multus-cni-dir\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.281173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.280116 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-run-k8s-cni-cncf-io\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.281967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.280233 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-dbus\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:55.281967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.280234 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-multus-cni-dir\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.281967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.280339 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-kubelet-config\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:55.281967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.280413 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-lib-modules\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.281967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.280450 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.281967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.280473 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cef09afa-b154-4fb4-b2cb-bcc07a390e40-iptables-alerter-script\") pod \"iptables-alerter-pmkbv\" (UID: \"cef09afa-b154-4fb4-b2cb-bcc07a390e40\") " pod="openshift-network-operator/iptables-alerter-pmkbv" Sep 15 11:36:55.281967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.280475 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b6f665e-7431-4599-b9ef-302eb545b1c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.281967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.280527 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9935a605-d4d3-4471-9740-cb993a66fe98-host-var-lib-kubelet\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.281967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.280574 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-sys\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.283421 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.282975 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7787af06-5984-4095-ac98-a920406f8b40-agent-certs\") pod \"konnectivity-agent-xqnpw\" (UID: \"7787af06-5984-4095-ac98-a920406f8b40\") " pod="kube-system/konnectivity-agent-xqnpw" Sep 15 11:36:55.283421 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.283007 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-tmp\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.284861 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.284839 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-etc-tuned\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.288437 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.288408 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:36:55.288437 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.288430 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:36:55.288437 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.288441 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6jxf9 for pod openshift-network-diagnostics/network-check-target-zdv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:36:55.288642 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.288488 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9 podName:ffa5fd6d-96fc-4cd4-ad93-269a473cd51b nodeName:}" failed. No retries permitted until 2026-09-15 11:36:55.78847501 +0000 UTC m=+3.018401524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6jxf9" (UniqueName: "kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9") pod "network-check-target-zdv99" (UID: "ffa5fd6d-96fc-4cd4-ad93-269a473cd51b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:36:55.289641 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.289620 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvq4t\" (UniqueName: \"kubernetes.io/projected/9935a605-d4d3-4471-9740-cb993a66fe98-kube-api-access-vvq4t\") pod \"multus-c9qbq\" (UID: \"9935a605-d4d3-4471-9740-cb993a66fe98\") " pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.290492 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.290375 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8229p\" (UniqueName: \"kubernetes.io/projected/cef09afa-b154-4fb4-b2cb-bcc07a390e40-kube-api-access-8229p\") pod \"iptables-alerter-pmkbv\" (UID: \"cef09afa-b154-4fb4-b2cb-bcc07a390e40\") " pod="openshift-network-operator/iptables-alerter-pmkbv" Sep 15 11:36:55.291386 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.291349 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k92tx\" (UniqueName: \"kubernetes.io/projected/05a4a7c8-24c6-4ac1-86f1-01c5ac17b444-kube-api-access-k92tx\") pod \"tuned-tdfp5\" (UID: \"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444\") " pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.291654 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.291637 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6w4\" (UniqueName: \"kubernetes.io/projected/91f95d2b-d44c-4f8a-83bf-649275a608e7-kube-api-access-8m6w4\") pod \"network-metrics-daemon-ldmmr\" (UID: \"91f95d2b-d44c-4f8a-83bf-649275a608e7\") " pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:36:55.292366 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.292346 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqbk\" (UniqueName: \"kubernetes.io/projected/4b6f665e-7431-4599-b9ef-302eb545b1c9-kube-api-access-khqbk\") pod \"multus-additional-cni-plugins-m4mzd\" (UID: \"4b6f665e-7431-4599-b9ef-302eb545b1c9\") " pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.293181 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.293138 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fkjd\" (UniqueName: \"kubernetes.io/projected/d5fd2c36-fb9a-43c3-9acb-f0f2088564c9-kube-api-access-8fkjd\") pod \"aws-ebs-csi-driver-node-rwmd9\" (UID: \"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.380369 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380329 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-slash\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.380369 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-run-netns\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.380601 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380400 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-systemd-units\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.380601 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.380601 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380444 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b4cec296-04c2-48ae-8024-09e47e456b59-hosts-file\") pod \"node-resolver-z8grp\" (UID: \"b4cec296-04c2-48ae-8024-09e47e456b59\") " pod="openshift-dns/node-resolver-z8grp" Sep 15 11:36:55.380601 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380464 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-slash\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.380601 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380480 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/087f48fe-aefc-4e62-b27f-d690c27cb027-env-overrides\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.380601 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380478 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-systemd-units\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.380601 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380500 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14fd92d1-ab33-45f3-b40f-10048dda0c33-serviceca\") pod \"node-ca-2sll8\" (UID: \"14fd92d1-ab33-45f3-b40f-10048dda0c33\") " pod="openshift-image-registry/node-ca-2sll8" Sep 15 11:36:55.380601 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380519 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-run-netns\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.380601 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380538 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.380601 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380560 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b4cec296-04c2-48ae-8024-09e47e456b59-hosts-file\") pod \"node-resolver-z8grp\" (UID: \"b4cec296-04c2-48ae-8024-09e47e456b59\") " pod="openshift-dns/node-resolver-z8grp" Sep 15 11:36:55.380601 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-kubelet\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380608 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-kubelet\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-var-lib-openvswitch\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380635 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-log-socket\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380662 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-run-ovn-kubernetes\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380685 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-cni-bin\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-cni-netd\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380724 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-run-ovn-kubernetes\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380725 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-var-lib-openvswitch\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/087f48fe-aefc-4e62-b27f-d690c27cb027-ovnkube-config\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380756 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-cni-netd\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380783 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-host-cni-bin\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380781 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkfxt\" (UniqueName: \"kubernetes.io/projected/b4cec296-04c2-48ae-8024-09e47e456b59-kube-api-access-fkfxt\") pod \"node-resolver-z8grp\" (UID: \"b4cec296-04c2-48ae-8024-09e47e456b59\") " pod="openshift-dns/node-resolver-z8grp" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380805 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14fd92d1-ab33-45f3-b40f-10048dda0c33-host\") pod \"node-ca-2sll8\" (UID: \"14fd92d1-ab33-45f3-b40f-10048dda0c33\") " pod="openshift-image-registry/node-ca-2sll8" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-run-ovn\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380847 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-node-log\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-run-systemd\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-etc-openvswitch\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/087f48fe-aefc-4e62-b27f-d690c27cb027-ovnkube-script-lib\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cccg\" (UniqueName: \"kubernetes.io/projected/087f48fe-aefc-4e62-b27f-d690c27cb027-kube-api-access-9cccg\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srs6s\" (UniqueName: \"kubernetes.io/projected/14fd92d1-ab33-45f3-b40f-10048dda0c33-kube-api-access-srs6s\") pod \"node-ca-2sll8\" (UID: \"14fd92d1-ab33-45f3-b40f-10048dda0c33\") " pod="openshift-image-registry/node-ca-2sll8" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-run-openvswitch\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380962 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/087f48fe-aefc-4e62-b27f-d690c27cb027-ovn-node-metrics-cert\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380972 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14fd92d1-ab33-45f3-b40f-10048dda0c33-serviceca\") pod \"node-ca-2sll8\" (UID: \"14fd92d1-ab33-45f3-b40f-10048dda0c33\") " pod="openshift-image-registry/node-ca-2sll8" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380751 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-log-socket\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.380977 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4cec296-04c2-48ae-8024-09e47e456b59-tmp-dir\") pod \"node-resolver-z8grp\" (UID: \"b4cec296-04c2-48ae-8024-09e47e456b59\") " pod="openshift-dns/node-resolver-z8grp" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.381002 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/087f48fe-aefc-4e62-b27f-d690c27cb027-env-overrides\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.381057 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-etc-openvswitch\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.381094 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/087f48fe-aefc-4e62-b27f-d690c27cb027-ovnkube-config\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.381100 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-run-ovn\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.381130 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14fd92d1-ab33-45f3-b40f-10048dda0c33-host\") pod \"node-ca-2sll8\" (UID: \"14fd92d1-ab33-45f3-b40f-10048dda0c33\") " pod="openshift-image-registry/node-ca-2sll8" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.381140 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-run-openvswitch\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.381100 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-run-systemd\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.381152 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/087f48fe-aefc-4e62-b27f-d690c27cb027-node-log\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.381181 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4cec296-04c2-48ae-8024-09e47e456b59-tmp-dir\") pod \"node-resolver-z8grp\" (UID: \"b4cec296-04c2-48ae-8024-09e47e456b59\") " pod="openshift-dns/node-resolver-z8grp" Sep 15 11:36:55.381808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.381549 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/087f48fe-aefc-4e62-b27f-d690c27cb027-ovnkube-script-lib\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.383652 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.383629 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/087f48fe-aefc-4e62-b27f-d690c27cb027-ovn-node-metrics-cert\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.391427 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.391367 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkfxt\" (UniqueName: \"kubernetes.io/projected/b4cec296-04c2-48ae-8024-09e47e456b59-kube-api-access-fkfxt\") pod \"node-resolver-z8grp\" (UID: \"b4cec296-04c2-48ae-8024-09e47e456b59\") " pod="openshift-dns/node-resolver-z8grp" Sep 15 11:36:55.391574 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.391551 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cccg\" (UniqueName: \"kubernetes.io/projected/087f48fe-aefc-4e62-b27f-d690c27cb027-kube-api-access-9cccg\") pod \"ovnkube-node-kj4f2\" (UID: \"087f48fe-aefc-4e62-b27f-d690c27cb027\") " pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.393529 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.393511 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srs6s\" (UniqueName: \"kubernetes.io/projected/14fd92d1-ab33-45f3-b40f-10048dda0c33-kube-api-access-srs6s\") pod \"node-ca-2sll8\" (UID: \"14fd92d1-ab33-45f3-b40f-10048dda0c33\") " pod="openshift-image-registry/node-ca-2sll8" Sep 15 11:36:55.437256 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.437229 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 11:36:55.466521 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.466485 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pmkbv" Sep 15 11:36:55.475093 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.475072 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m4mzd" Sep 15 11:36:55.485978 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.485957 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xqnpw" Sep 15 11:36:55.490685 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.490662 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" Sep 15 11:36:55.498283 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.498262 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" Sep 15 11:36:55.506857 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.506835 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c9qbq" Sep 15 11:36:55.514561 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.514539 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:36:55.521115 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.521094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z8grp" Sep 15 11:36:55.526826 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.526803 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2sll8" Sep 15 11:36:55.783859 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.783830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:55.784046 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.783888 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs\") pod \"network-metrics-daemon-ldmmr\" (UID: \"91f95d2b-d44c-4f8a-83bf-649275a608e7\") " pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:36:55.784046 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.783986 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:36:55.784141 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.784059 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret podName:ecdd2ee0-6fba-4398-ad1b-87c1a4dca820 nodeName:}" failed. No retries permitted until 2026-09-15 11:36:56.784044662 +0000 UTC m=+4.013971180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret") pod "global-pull-secret-syncer-msl5t" (UID: "ecdd2ee0-6fba-4398-ad1b-87c1a4dca820") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:36:55.784141 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.783986 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:36:55.784141 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.784115 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs podName:91f95d2b-d44c-4f8a-83bf-649275a608e7 nodeName:}" failed. No retries permitted until 2026-09-15 11:36:56.784103334 +0000 UTC m=+4.014029863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs") pod "network-metrics-daemon-ldmmr" (UID: "91f95d2b-d44c-4f8a-83bf-649275a608e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:36:55.884642 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:55.884607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxf9\" (UniqueName: \"kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9\") pod \"network-check-target-zdv99\" (UID: \"ffa5fd6d-96fc-4cd4-ad93-269a473cd51b\") " pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:36:55.884803 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.884769 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:36:55.884803 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.884790 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:36:55.884803 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.884799 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6jxf9 for pod openshift-network-diagnostics/network-check-target-zdv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:36:55.884946 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:55.884855 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9 podName:ffa5fd6d-96fc-4cd4-ad93-269a473cd51b nodeName:}" failed. No retries permitted until 2026-09-15 11:36:56.8848381 +0000 UTC m=+4.114764616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jxf9" (UniqueName: "kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9") pod "network-check-target-zdv99" (UID: "ffa5fd6d-96fc-4cd4-ad93-269a473cd51b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:36:56.208793 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:56.208705 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:31:55 +0000 UTC" deadline="2028-03-27 01:30:11.403568333 +0000 UTC" Sep 15 11:36:56.208793 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:56.208751 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13405h53m15.194821046s" Sep 15 11:36:56.235165 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:56.235137 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087f48fe_aefc_4e62_b27f_d690c27cb027.slice/crio-258fea5cf2b4aabefbc861426013aced487effa742e7f280582d33d2086551f0 WatchSource:0}: Error finding container 258fea5cf2b4aabefbc861426013aced487effa742e7f280582d33d2086551f0: Status 404 returned error can't find the container with id 258fea5cf2b4aabefbc861426013aced487effa742e7f280582d33d2086551f0 Sep 15 11:36:56.236650 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:56.236619 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6f665e_7431_4599_b9ef_302eb545b1c9.slice/crio-17d2b670d2c6b26953cf9b98161883dcf1f64f9714254a7ffa395d4bfe57663a WatchSource:0}: Error finding container 17d2b670d2c6b26953cf9b98161883dcf1f64f9714254a7ffa395d4bfe57663a: Status 404 returned error can't find the container with id 17d2b670d2c6b26953cf9b98161883dcf1f64f9714254a7ffa395d4bfe57663a Sep 15 11:36:56.238131 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:56.238089 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" event={"ID":"087f48fe-aefc-4e62-b27f-d690c27cb027","Type":"ContainerStarted","Data":"258fea5cf2b4aabefbc861426013aced487effa742e7f280582d33d2086551f0"} Sep 15 11:36:56.238422 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:56.238373 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cec296_04c2_48ae_8024_09e47e456b59.slice/crio-d02d52933f2c62ee6a45bb03ca042666373b2ad60e61be8da7e9aa2b7b82ed0a WatchSource:0}: Error finding container d02d52933f2c62ee6a45bb03ca042666373b2ad60e61be8da7e9aa2b7b82ed0a: Status 404 returned error can't find the container with id d02d52933f2c62ee6a45bb03ca042666373b2ad60e61be8da7e9aa2b7b82ed0a Sep 15 11:36:56.241566 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:56.241503 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fd2c36_fb9a_43c3_9acb_f0f2088564c9.slice/crio-ceb12227a42475dde3dc2cb8d2f8de9d86047abbc2ab412870fa56433bcea73c WatchSource:0}: Error finding container ceb12227a42475dde3dc2cb8d2f8de9d86047abbc2ab412870fa56433bcea73c: Status 404 returned error can't find the container with id ceb12227a42475dde3dc2cb8d2f8de9d86047abbc2ab412870fa56433bcea73c Sep 15 11:36:56.241988 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:56.241968 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7787af06_5984_4095_ac98_a920406f8b40.slice/crio-8e27929ffbdfedd002da2f2fa955a2dfa7747e13c93056197960f67a02f3186c WatchSource:0}: Error finding container 8e27929ffbdfedd002da2f2fa955a2dfa7747e13c93056197960f67a02f3186c: Status 404 returned error can't find the container with id 8e27929ffbdfedd002da2f2fa955a2dfa7747e13c93056197960f67a02f3186c Sep 15 11:36:56.243398 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:56.243321 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef09afa_b154_4fb4_b2cb_bcc07a390e40.slice/crio-c8e6fe73f05e1de092c85536cde256e48f0339df9f58dc3f2909ef3c8f05867e WatchSource:0}: Error finding container c8e6fe73f05e1de092c85536cde256e48f0339df9f58dc3f2909ef3c8f05867e: Status 404 returned error can't find the container with id c8e6fe73f05e1de092c85536cde256e48f0339df9f58dc3f2909ef3c8f05867e Sep 15 11:36:56.244519 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:56.244313 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fd92d1_ab33_45f3_b40f_10048dda0c33.slice/crio-53b6d6163022a39e9b5a7df8e0b1d23695f84a6be962212e3a15cf83f9408d48 WatchSource:0}: Error finding container 53b6d6163022a39e9b5a7df8e0b1d23695f84a6be962212e3a15cf83f9408d48: Status 404 returned error can't find the container with id 53b6d6163022a39e9b5a7df8e0b1d23695f84a6be962212e3a15cf83f9408d48 Sep 15 11:36:56.246296 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:36:56.246203 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9935a605_d4d3_4471_9740_cb993a66fe98.slice/crio-bafa57704ee3d554ab1af94f08df53e57903c52ab779b1b548108f7b6e9218f1 WatchSource:0}: Error finding container bafa57704ee3d554ab1af94f08df53e57903c52ab779b1b548108f7b6e9218f1: Status 404 returned error can't find the container with id bafa57704ee3d554ab1af94f08df53e57903c52ab779b1b548108f7b6e9218f1 Sep 15 11:36:56.792429 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:56.791634 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs\") pod \"network-metrics-daemon-ldmmr\" (UID: \"91f95d2b-d44c-4f8a-83bf-649275a608e7\") " pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:36:56.792429 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:56.791696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:56.792429 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:56.791864 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:36:56.792429 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:56.791925 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret podName:ecdd2ee0-6fba-4398-ad1b-87c1a4dca820 nodeName:}" failed. No retries permitted until 2026-09-15 11:36:58.791906027 +0000 UTC m=+6.021832544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret") pod "global-pull-secret-syncer-msl5t" (UID: "ecdd2ee0-6fba-4398-ad1b-87c1a4dca820") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:36:56.792429 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:56.792346 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:36:56.792429 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:56.792396 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs podName:91f95d2b-d44c-4f8a-83bf-649275a608e7 nodeName:}" failed. No retries permitted until 2026-09-15 11:36:58.792381368 +0000 UTC m=+6.022307885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs") pod "network-metrics-daemon-ldmmr" (UID: "91f95d2b-d44c-4f8a-83bf-649275a608e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:36:56.893047 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:56.892459 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxf9\" (UniqueName: \"kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9\") pod \"network-check-target-zdv99\" (UID: \"ffa5fd6d-96fc-4cd4-ad93-269a473cd51b\") " pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:36:56.893047 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:56.892601 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:36:56.893047 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:56.892620 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:36:56.893047 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:56.892633 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6jxf9 for pod openshift-network-diagnostics/network-check-target-zdv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:36:56.893047 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:56.892686 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9 podName:ffa5fd6d-96fc-4cd4-ad93-269a473cd51b nodeName:}" failed. No retries permitted until 2026-09-15 11:36:58.892668058 +0000 UTC m=+6.122594573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jxf9" (UniqueName: "kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9") pod "network-check-target-zdv99" (UID: "ffa5fd6d-96fc-4cd4-ad93-269a473cd51b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:36:56.962759 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:56.962509 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 11:36:57.209709 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.209589 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:31:55 +0000 UTC" deadline="2028-06-19 16:36:12.93914202 +0000 UTC" Sep 15 11:36:57.209709 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.209623 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15436h59m15.729522843s" Sep 15 11:36:57.237796 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.237633 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:36:57.237796 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:57.237789 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:36:57.238317 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.238290 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:57.238443 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:57.238425 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:36:57.239088 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.238523 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:36:57.239088 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:57.238662 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:36:57.246289 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.246259 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pmkbv" event={"ID":"cef09afa-b154-4fb4-b2cb-bcc07a390e40","Type":"ContainerStarted","Data":"c8e6fe73f05e1de092c85536cde256e48f0339df9f58dc3f2909ef3c8f05867e"} Sep 15 11:36:57.249050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.248205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" event={"ID":"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9","Type":"ContainerStarted","Data":"ceb12227a42475dde3dc2cb8d2f8de9d86047abbc2ab412870fa56433bcea73c"} Sep 15 11:36:57.249789 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.249766 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4mzd" event={"ID":"4b6f665e-7431-4599-b9ef-302eb545b1c9","Type":"ContainerStarted","Data":"17d2b670d2c6b26953cf9b98161883dcf1f64f9714254a7ffa395d4bfe57663a"} Sep 15 11:36:57.251194 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.251170 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2sll8" event={"ID":"14fd92d1-ab33-45f3-b40f-10048dda0c33","Type":"ContainerStarted","Data":"53b6d6163022a39e9b5a7df8e0b1d23695f84a6be962212e3a15cf83f9408d48"} Sep 15 11:36:57.254921 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.254896 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c9qbq" event={"ID":"9935a605-d4d3-4471-9740-cb993a66fe98","Type":"ContainerStarted","Data":"bafa57704ee3d554ab1af94f08df53e57903c52ab779b1b548108f7b6e9218f1"} Sep 15 11:36:57.258485 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.258462 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z8grp" event={"ID":"b4cec296-04c2-48ae-8024-09e47e456b59","Type":"ContainerStarted","Data":"d02d52933f2c62ee6a45bb03ca042666373b2ad60e61be8da7e9aa2b7b82ed0a"} Sep 15 11:36:57.282485 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.282458 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-189.ec2.internal" event={"ID":"0cce461e99c8c476d4a7a451016bffc0","Type":"ContainerStarted","Data":"bfdad9d68cec2c0274a1c66304425b89cf0ecde3de4272715d6bb57d84f41588"} Sep 15 11:36:57.288214 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.288188 2573 generic.go:356] "Generic (PLEG): container finished" podID="f8e4755d9e46fb80d2e9e4a381fbae35" containerID="8c5a907ce49781ea15e3becd6385a6b7764c692c9e60a866801c53fc47d04965" exitCode=0 Sep 15 11:36:57.288353 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.288269 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal" event={"ID":"f8e4755d9e46fb80d2e9e4a381fbae35","Type":"ContainerDied","Data":"8c5a907ce49781ea15e3becd6385a6b7764c692c9e60a866801c53fc47d04965"} Sep 15 11:36:57.298171 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.298122 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xqnpw" event={"ID":"7787af06-5984-4095-ac98-a920406f8b40","Type":"ContainerStarted","Data":"8e27929ffbdfedd002da2f2fa955a2dfa7747e13c93056197960f67a02f3186c"} Sep 15 11:36:57.300098 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.298608 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-189.ec2.internal" podStartSLOduration=3.298594221 podStartE2EDuration="3.298594221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:36:57.298418298 +0000 UTC m=+4.528344835" watchObservedRunningTime="2026-09-15 11:36:57.298594221 +0000 UTC m=+4.528520758" Sep 15 11:36:57.304500 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:57.304444 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" event={"ID":"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444","Type":"ContainerStarted","Data":"411dbff57ff952d74d2b730f6e4ad644d7a642a46366364663aab8f4d9c6433e"} Sep 15 11:36:58.328296 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:58.328125 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal" event={"ID":"f8e4755d9e46fb80d2e9e4a381fbae35","Type":"ContainerStarted","Data":"fcf104d4ab14e6c94854e00a5be2c696d832f75f2bfb0682f8642629f2829204"} Sep 15 11:36:58.809221 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:58.809188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs\") pod \"network-metrics-daemon-ldmmr\" (UID: \"91f95d2b-d44c-4f8a-83bf-649275a608e7\") " pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:36:58.809417 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:58.809249 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:58.809417 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:58.809383 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:36:58.809555 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:58.809429 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:36:58.809555 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:58.809452 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret podName:ecdd2ee0-6fba-4398-ad1b-87c1a4dca820 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:02.809430318 +0000 UTC m=+10.039356846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret") pod "global-pull-secret-syncer-msl5t" (UID: "ecdd2ee0-6fba-4398-ad1b-87c1a4dca820") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:36:58.809555 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:58.809492 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs podName:91f95d2b-d44c-4f8a-83bf-649275a608e7 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:02.809472839 +0000 UTC m=+10.039399392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs") pod "network-metrics-daemon-ldmmr" (UID: "91f95d2b-d44c-4f8a-83bf-649275a608e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:36:58.909837 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:58.909803 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxf9\" (UniqueName: \"kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9\") pod \"network-check-target-zdv99\" (UID: \"ffa5fd6d-96fc-4cd4-ad93-269a473cd51b\") " pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:36:58.910039 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:58.910002 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:36:58.910119 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:58.910045 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:36:58.910119 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:58.910059 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6jxf9 for pod openshift-network-diagnostics/network-check-target-zdv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:36:58.910228 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:58.910119 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9 podName:ffa5fd6d-96fc-4cd4-ad93-269a473cd51b nodeName:}" failed. No retries permitted until 2026-09-15 11:37:02.910099357 +0000 UTC m=+10.140025885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jxf9" (UniqueName: "kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9") pod "network-check-target-zdv99" (UID: "ffa5fd6d-96fc-4cd4-ad93-269a473cd51b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:36:59.233430 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:59.233353 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:36:59.233573 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:59.233479 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:36:59.233884 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:59.233860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:36:59.233997 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:59.233964 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:36:59.234091 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:36:59.234042 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:36:59.234144 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:36:59.234131 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:01.229744 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:01.229008 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:01.229744 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:01.229069 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:01.229744 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:01.229152 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:01.229744 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:01.229245 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:01.229744 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:01.229288 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:01.229744 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:01.229372 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:02.848511 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:02.848466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:02.848989 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:02.848558 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs\") pod \"network-metrics-daemon-ldmmr\" (UID: \"91f95d2b-d44c-4f8a-83bf-649275a608e7\") " pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:02.848989 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:02.848632 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:37:02.848989 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:02.848676 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:37:02.848989 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:02.848711 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret podName:ecdd2ee0-6fba-4398-ad1b-87c1a4dca820 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:10.848691474 +0000 UTC m=+18.078618005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret") pod "global-pull-secret-syncer-msl5t" (UID: "ecdd2ee0-6fba-4398-ad1b-87c1a4dca820") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:37:02.848989 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:02.848733 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs podName:91f95d2b-d44c-4f8a-83bf-649275a608e7 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:10.848722408 +0000 UTC m=+18.078648926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs") pod "network-metrics-daemon-ldmmr" (UID: "91f95d2b-d44c-4f8a-83bf-649275a608e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:37:02.949930 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:02.949612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxf9\" (UniqueName: \"kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9\") pod \"network-check-target-zdv99\" (UID: \"ffa5fd6d-96fc-4cd4-ad93-269a473cd51b\") " pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:02.949930 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:02.949849 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:37:02.949930 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:02.949871 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:37:02.949930 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:02.949883 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6jxf9 for pod openshift-network-diagnostics/network-check-target-zdv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:37:02.950296 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:02.949948 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9 podName:ffa5fd6d-96fc-4cd4-ad93-269a473cd51b nodeName:}" failed. No retries permitted until 2026-09-15 11:37:10.949930209 +0000 UTC m=+18.179856722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jxf9" (UniqueName: "kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9") pod "network-check-target-zdv99" (UID: "ffa5fd6d-96fc-4cd4-ad93-269a473cd51b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:37:03.232805 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:03.232731 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:03.232958 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:03.232838 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:03.232958 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:03.232853 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:03.232958 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:03.232833 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:03.233108 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:03.232940 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:03.233108 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:03.233054 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:05.228904 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:05.228871 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:05.229385 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:05.228916 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:05.229385 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:05.228995 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:05.229385 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:05.229143 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:05.229385 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:05.229241 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:05.229385 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:05.229340 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:07.229341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:07.229300 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:07.229812 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:07.229299 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:07.229812 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:07.229435 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:07.229812 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:07.229441 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:07.229812 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:07.229534 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:07.229812 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:07.229619 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:09.232500 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:09.232467 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:09.232500 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:09.232483 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:09.232988 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:09.232475 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:09.232988 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:09.232610 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:09.232988 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:09.232739 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:09.232988 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:09.232819 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:10.902459 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:10.902409 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs\") pod \"network-metrics-daemon-ldmmr\" (UID: \"91f95d2b-d44c-4f8a-83bf-649275a608e7\") " pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:10.902919 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:10.902489 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:10.902919 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:10.902541 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:37:10.902919 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:10.902610 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:37:10.902919 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:10.902612 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs podName:91f95d2b-d44c-4f8a-83bf-649275a608e7 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:26.902592776 +0000 UTC m=+34.132519304 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs") pod "network-metrics-daemon-ldmmr" (UID: "91f95d2b-d44c-4f8a-83bf-649275a608e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:37:10.902919 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:10.902703 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret podName:ecdd2ee0-6fba-4398-ad1b-87c1a4dca820 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:26.902660026 +0000 UTC m=+34.132586543 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret") pod "global-pull-secret-syncer-msl5t" (UID: "ecdd2ee0-6fba-4398-ad1b-87c1a4dca820") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:37:11.003533 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:11.003498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxf9\" (UniqueName: \"kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9\") pod \"network-check-target-zdv99\" (UID: \"ffa5fd6d-96fc-4cd4-ad93-269a473cd51b\") " pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:11.003731 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:11.003642 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:37:11.003731 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:11.003668 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:37:11.003731 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:11.003683 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6jxf9 for pod openshift-network-diagnostics/network-check-target-zdv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:37:11.003878 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:11.003749 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9 podName:ffa5fd6d-96fc-4cd4-ad93-269a473cd51b nodeName:}" failed. No retries permitted until 2026-09-15 11:37:27.003731625 +0000 UTC m=+34.233658155 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jxf9" (UniqueName: "kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9") pod "network-check-target-zdv99" (UID: "ffa5fd6d-96fc-4cd4-ad93-269a473cd51b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:37:11.229103 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:11.229013 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:11.229103 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:11.229038 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:11.229319 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:11.229013 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:11.229319 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:11.229158 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:11.229319 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:11.229221 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:11.229319 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:11.229276 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:13.231957 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:13.231924 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:13.231957 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:13.231938 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:13.232501 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:13.231923 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:13.232501 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:13.232046 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:13.232501 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:13.232126 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:13.232501 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:13.232199 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:14.365936 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:14.365712 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xqnpw" event={"ID":"7787af06-5984-4095-ac98-a920406f8b40","Type":"ContainerStarted","Data":"2b9f66f4b0821070844f3e4b234fa8378f6063f26bcd391ae680e62b843e7646"} Sep 15 11:37:14.367408 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:14.367376 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" event={"ID":"05a4a7c8-24c6-4ac1-86f1-01c5ac17b444","Type":"ContainerStarted","Data":"a7e96cf523789020ca3ba8141953298f3a315cb68f3de617bb6c2d37a7e4926a"} Sep 15 11:37:14.368965 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:14.368938 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" event={"ID":"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9","Type":"ContainerStarted","Data":"39551bda7bb297b3f9c17fbc885f1deab28e1060ec66a762061f088b46abc43d"} Sep 15 11:37:14.370725 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:14.370702 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4mzd" event={"ID":"4b6f665e-7431-4599-b9ef-302eb545b1c9","Type":"ContainerStarted","Data":"69a9d01f14360238c7209a266b1acaf8a3ecedc2cef89dfb73bfeb516e7374c9"} Sep 15 11:37:14.372242 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:14.372221 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2sll8" event={"ID":"14fd92d1-ab33-45f3-b40f-10048dda0c33","Type":"ContainerStarted","Data":"61ed41df20623e0d9fe56d05abb7461268df769cd3a2ae5d2fbead4263503145"} Sep 15 11:37:14.373879 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:14.373846 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c9qbq" event={"ID":"9935a605-d4d3-4471-9740-cb993a66fe98","Type":"ContainerStarted","Data":"3615e1e6d3fd738408b6335f7d589377ce2b32b8ff58cceb63f2d41ff420ea78"} Sep 15 11:37:14.375484 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:14.375462 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z8grp" event={"ID":"b4cec296-04c2-48ae-8024-09e47e456b59","Type":"ContainerStarted","Data":"f7de6c6f4855635da0fa0cc8030fb64785a73b74a95f85c359d0a9c0e69042d8"} Sep 15 11:37:14.383202 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:14.383163 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-189.ec2.internal" podStartSLOduration=20.383151076 podStartE2EDuration="20.383151076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:36:58.346253587 +0000 UTC m=+5.576180124" watchObservedRunningTime="2026-09-15 11:37:14.383151076 +0000 UTC m=+21.613077611" Sep 15 11:37:14.400361 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:14.400308 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xqnpw" podStartSLOduration=3.748519312 podStartE2EDuration="21.400288655s" totalImagesPullingTime="17.651769343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:14.383216639 +0000 UTC m=+21.613143192" watchObservedRunningTime="2026-09-15 11:37:14.400288655 +0000 UTC m=+21.630215201" Sep 15 11:37:14.401005 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:14.400968 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z8grp" podStartSLOduration=3.6817798699999997 podStartE2EDuration="21.400957369s" totalImagesPullingTime="17.719177499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:14.399889089 +0000 UTC m=+21.629815626" watchObservedRunningTime="2026-09-15 11:37:14.400957369 +0000 UTC m=+21.630883905" Sep 15 11:37:14.439124 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:14.439082 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tdfp5" podStartSLOduration=3.725848575 podStartE2EDuration="21.439067243s" totalImagesPullingTime="17.713218668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:14.43891076 +0000 UTC m=+21.668837297" watchObservedRunningTime="2026-09-15 11:37:14.439067243 +0000 UTC m=+21.668993778" Sep 15 11:37:14.458414 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:14.458364 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c9qbq" podStartSLOduration=3.736973018 podStartE2EDuration="21.458349657s" totalImagesPullingTime="17.721376639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:14.458057844 +0000 UTC m=+21.687984381" watchObservedRunningTime="2026-09-15 11:37:14.458349657 +0000 UTC m=+21.688276193" Sep 15 11:37:15.229123 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.228937 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:15.229239 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.228952 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:15.229298 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:15.229228 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:15.229298 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.228952 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:15.229298 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:15.229268 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:15.229442 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:15.229343 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:15.380480 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.380416 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 11:37:15.381232 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.380787 2573 generic.go:356] "Generic (PLEG): container finished" podID="087f48fe-aefc-4e62-b27f-d690c27cb027" containerID="0c87d9c130cfc71f0d34fddbb0f1d5ee32fbf69c7b965ea040cad1531e5a1fae" exitCode=1 Sep 15 11:37:15.381232 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.380853 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" event={"ID":"087f48fe-aefc-4e62-b27f-d690c27cb027","Type":"ContainerStarted","Data":"fc8706ac4c854c454375dc6486546988e9d8d954c0bfcb5c0bd735cfb7d28a08"} Sep 15 11:37:15.381232 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.380920 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" event={"ID":"087f48fe-aefc-4e62-b27f-d690c27cb027","Type":"ContainerStarted","Data":"aaeb069153bfd36b96d3df30d363b3098ba1421f0f0b26a0d7ad55041e59ff62"} Sep 15 11:37:15.381232 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.380954 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" event={"ID":"087f48fe-aefc-4e62-b27f-d690c27cb027","Type":"ContainerStarted","Data":"229e3972dda16165a539f1fb086f4f85f86e9a2474738f44c4d7d8e06f0117d4"} Sep 15 11:37:15.381232 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.380989 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" event={"ID":"087f48fe-aefc-4e62-b27f-d690c27cb027","Type":"ContainerStarted","Data":"bde09371d171eb7a53d5bad39d79bbbb6272cc1ca7fee850d7fe98793fdb85b0"} Sep 15 11:37:15.381232 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.381017 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" event={"ID":"087f48fe-aefc-4e62-b27f-d690c27cb027","Type":"ContainerDied","Data":"0c87d9c130cfc71f0d34fddbb0f1d5ee32fbf69c7b965ea040cad1531e5a1fae"} Sep 15 11:37:15.381232 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.381067 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" event={"ID":"087f48fe-aefc-4e62-b27f-d690c27cb027","Type":"ContainerStarted","Data":"98641e02a19db89dcc4afd236ac8d6814e45949c82df453841134a1120d0b9e2"} Sep 15 11:37:15.383758 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.383738 2573 generic.go:356] "Generic (PLEG): container finished" podID="4b6f665e-7431-4599-b9ef-302eb545b1c9" containerID="69a9d01f14360238c7209a266b1acaf8a3ecedc2cef89dfb73bfeb516e7374c9" exitCode=0 Sep 15 11:37:15.383856 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.383835 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4mzd" event={"ID":"4b6f665e-7431-4599-b9ef-302eb545b1c9","Type":"ContainerDied","Data":"69a9d01f14360238c7209a266b1acaf8a3ecedc2cef89dfb73bfeb516e7374c9"} Sep 15 11:37:15.406201 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.406160 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2sll8" podStartSLOduration=4.6939899480000005 podStartE2EDuration="22.406147995s" totalImagesPullingTime="17.712158047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:14.476159483 +0000 UTC m=+21.706086001" watchObservedRunningTime="2026-09-15 11:37:15.406147995 +0000 UTC m=+22.636074531" Sep 15 11:37:15.552333 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:15.552312 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 15 11:37:16.224668 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:16.224511 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T11:37:15.552327334Z","UUID":"3a95f98b-4e09-42aa-9a44-934e4af70d21","Handler":null,"Name":"","Endpoint":""} Sep 15 11:37:16.226384 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:16.226361 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 15 11:37:16.226519 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:16.226398 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 15 11:37:16.387373 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:16.387336 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pmkbv" event={"ID":"cef09afa-b154-4fb4-b2cb-bcc07a390e40","Type":"ContainerStarted","Data":"a62bd33bf9810cc7aacb9050ea6a29c0bac97df2b587d46574c6a4a3a80f3200"} Sep 15 11:37:16.389236 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:16.389205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" event={"ID":"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9","Type":"ContainerStarted","Data":"f5da91f836df428e25be1d5c7c215dc0a53824695de69daf7b43fb33d1945c70"} Sep 15 11:37:16.402631 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:16.402589 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pmkbv" podStartSLOduration=5.68263763 podStartE2EDuration="23.402575323s" totalImagesPullingTime="17.719937693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:16.402326457 +0000 UTC m=+23.632252995" watchObservedRunningTime="2026-09-15 11:37:16.402575323 +0000 UTC m=+23.632501859" Sep 15 11:37:17.229392 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:17.229363 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:17.229632 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:17.229363 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:17.229632 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:17.229466 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:17.229632 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:17.229369 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:17.229632 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:17.229536 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:17.229632 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:17.229615 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:17.395265 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:17.395244 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 11:37:17.399015 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:17.398988 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" event={"ID":"d5fd2c36-fb9a-43c3-9acb-f0f2088564c9","Type":"ContainerStarted","Data":"2f34470f32d752fb955d309df5a505c02a886a78af6106294cedd63d9a5a6284"} Sep 15 11:37:17.417749 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:17.417699 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rwmd9" podStartSLOduration=4.661792124 podStartE2EDuration="24.417638062s" totalImagesPullingTime="19.755845938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:36:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:17.417304194 +0000 UTC m=+24.647230770" watchObservedRunningTime="2026-09-15 11:37:17.417638062 +0000 UTC m=+24.647564599" Sep 15 11:37:18.406109 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:18.405923 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 11:37:18.406639 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:18.406613 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" event={"ID":"087f48fe-aefc-4e62-b27f-d690c27cb027","Type":"ContainerStarted","Data":"376dfb83d19bdf2afe93ec0c48dc03a4bedc1b08e28b95ed52984f86c837bf88"} Sep 15 11:37:19.060111 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:19.060080 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xqnpw" Sep 15 11:37:19.060711 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:19.060685 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xqnpw" Sep 15 11:37:19.229125 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:19.229102 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:19.229283 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:19.229219 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:19.229653 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:19.229591 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:19.229729 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:19.229703 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:19.229774 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:19.229745 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:19.229832 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:19.229814 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:19.409700 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:19.409632 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xqnpw" Sep 15 11:37:19.410292 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:19.410214 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xqnpw" Sep 15 11:37:20.416150 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:20.415931 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 11:37:20.416826 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:20.416477 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" event={"ID":"087f48fe-aefc-4e62-b27f-d690c27cb027","Type":"ContainerStarted","Data":"0ab1c4627124a1372bfde076a71d8691f2274549e7af729b414a15163c62e3cb"} Sep 15 11:37:20.416922 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:20.416905 2573 scope.go:117] "RemoveContainer" containerID="0c87d9c130cfc71f0d34fddbb0f1d5ee32fbf69c7b965ea040cad1531e5a1fae" Sep 15 11:37:21.228676 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:21.228650 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:21.228833 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:21.228655 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:21.228833 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:21.228743 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:21.228833 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:21.228818 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:21.228930 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:21.228871 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:21.229008 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:21.228970 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:21.420743 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:21.420668 2573 generic.go:356] "Generic (PLEG): container finished" podID="4b6f665e-7431-4599-b9ef-302eb545b1c9" containerID="fec18bf7cf3aa049b25ab68327d1f90293113dbc5b030f39dc274f9fa9d2f5fa" exitCode=0 Sep 15 11:37:21.421179 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:21.420749 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4mzd" event={"ID":"4b6f665e-7431-4599-b9ef-302eb545b1c9","Type":"ContainerDied","Data":"fec18bf7cf3aa049b25ab68327d1f90293113dbc5b030f39dc274f9fa9d2f5fa"} Sep 15 11:37:21.424695 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:21.424679 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 11:37:21.425151 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:21.425127 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" event={"ID":"087f48fe-aefc-4e62-b27f-d690c27cb027","Type":"ContainerStarted","Data":"7f80ed57f1c7c0573b04ccf6319c1079181415bd3b06dd618f3d892dd45ab4d9"} Sep 15 11:37:21.425448 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:21.425424 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:37:21.425603 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:21.425492 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:37:21.425603 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:21.425542 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:37:21.442440 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:21.442414 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:37:21.442592 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:21.442535 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:37:21.485726 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:21.485673 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" podStartSLOduration=10.291422484 podStartE2EDuration="28.485657767s" totalImagesPullingTime="18.194235283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:21.485156475 +0000 UTC m=+28.715083014" watchObservedRunningTime="2026-09-15 11:37:21.485657767 +0000 UTC m=+28.715584302" Sep 15 11:37:22.151281 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:22.151101 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-msl5t"] Sep 15 11:37:22.151281 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:22.151275 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:22.151470 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:22.151378 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:22.154848 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:22.154822 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zdv99"] Sep 15 11:37:22.154982 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:22.154934 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:22.155067 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:22.155036 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:22.167033 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:22.166991 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldmmr"] Sep 15 11:37:22.167159 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:22.167116 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:22.167230 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:22.167212 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:23.229828 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:23.229798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:23.230214 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:23.229880 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:23.431764 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:23.431723 2573 generic.go:356] "Generic (PLEG): container finished" podID="4b6f665e-7431-4599-b9ef-302eb545b1c9" containerID="aa707433be848f2db6596f8f741a18247e52c409f60f84f1d8effe1166f8fecd" exitCode=0 Sep 15 11:37:23.431925 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:23.431795 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4mzd" event={"ID":"4b6f665e-7431-4599-b9ef-302eb545b1c9","Type":"ContainerDied","Data":"aa707433be848f2db6596f8f741a18247e52c409f60f84f1d8effe1166f8fecd"} Sep 15 11:37:24.228539 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:24.228503 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:24.228683 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:24.228515 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:24.228683 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:24.228644 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:24.228765 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:24.228711 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:25.228998 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:25.228823 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:25.229395 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:25.229083 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:25.438562 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:25.438528 2573 generic.go:356] "Generic (PLEG): container finished" podID="4b6f665e-7431-4599-b9ef-302eb545b1c9" containerID="d196276808d436da57df3c70b13e8bf8cab4179b39aaceebb98ad1a9d0c8eec6" exitCode=0 Sep 15 11:37:25.438714 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:25.438577 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4mzd" event={"ID":"4b6f665e-7431-4599-b9ef-302eb545b1c9","Type":"ContainerDied","Data":"d196276808d436da57df3c70b13e8bf8cab4179b39aaceebb98ad1a9d0c8eec6"} Sep 15 11:37:26.228398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:26.228362 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:26.228564 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:26.228362 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:26.228564 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:26.228471 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdv99" podUID="ffa5fd6d-96fc-4cd4-ad93-269a473cd51b" Sep 15 11:37:26.228564 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:26.228553 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmmr" podUID="91f95d2b-d44c-4f8a-83bf-649275a608e7" Sep 15 11:37:26.924103 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:26.924066 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:26.924626 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:26.924155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs\") pod \"network-metrics-daemon-ldmmr\" (UID: \"91f95d2b-d44c-4f8a-83bf-649275a608e7\") " pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:26.924626 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:26.924232 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:37:26.924626 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:26.924273 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:37:26.924626 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:26.924311 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret podName:ecdd2ee0-6fba-4398-ad1b-87c1a4dca820 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:58.924291431 +0000 UTC m=+66.154217964 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret") pod "global-pull-secret-syncer-msl5t" (UID: "ecdd2ee0-6fba-4398-ad1b-87c1a4dca820") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:37:26.924626 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:26.924330 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs podName:91f95d2b-d44c-4f8a-83bf-649275a608e7 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:58.924321528 +0000 UTC m=+66.154248041 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs") pod "network-metrics-daemon-ldmmr" (UID: "91f95d2b-d44c-4f8a-83bf-649275a608e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:37:27.024481 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:27.024448 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxf9\" (UniqueName: \"kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9\") pod \"network-check-target-zdv99\" (UID: \"ffa5fd6d-96fc-4cd4-ad93-269a473cd51b\") " pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:27.024655 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:27.024635 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:37:27.024701 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:27.024658 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:37:27.024701 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:27.024668 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6jxf9 for pod openshift-network-diagnostics/network-check-target-zdv99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:37:27.024775 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:27.024729 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9 podName:ffa5fd6d-96fc-4cd4-ad93-269a473cd51b nodeName:}" failed. No retries permitted until 2026-09-15 11:37:59.02471358 +0000 UTC m=+66.254640108 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jxf9" (UniqueName: "kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9") pod "network-check-target-zdv99" (UID: "ffa5fd6d-96fc-4cd4-ad93-269a473cd51b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:37:27.228789 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:27.228706 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:27.228972 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:27.228839 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-msl5t" podUID="ecdd2ee0-6fba-4398-ad1b-87c1a4dca820" Sep 15 11:37:28.167883 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.167849 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-189.ec2.internal" event="NodeReady" Sep 15 11:37:28.168516 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.168011 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 15 11:37:28.204874 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.204841 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5"] Sep 15 11:37:28.224267 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.224242 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cf4d4794-zt4c6"] Sep 15 11:37:28.224429 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.224385 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:28.228624 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.228432 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 15 11:37:28.228624 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.228472 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 15 11:37:28.228831 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.228776 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 15 11:37:28.229427 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.229402 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m5mwt\"" Sep 15 11:37:28.231445 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.231422 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 15 11:37:28.240415 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.240390 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-hd9fz"] Sep 15 11:37:28.240528 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.240480 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.241342 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.240481 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:28.241342 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.240481 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:28.243226 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.243196 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 15 11:37:28.243332 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.243245 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 11:37:28.243470 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.243452 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 15 11:37:28.243564 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.243550 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 11:37:28.243778 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.243662 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-72bfh\"" Sep 15 11:37:28.243879 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.243836 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 15 11:37:28.244032 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.243992 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ctgh2\"" Sep 15 11:37:28.244149 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.244130 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w6k5p\"" Sep 15 11:37:28.244492 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.244357 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 11:37:28.250844 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.250821 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 15 11:37:28.260545 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.260520 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5ddcf9f498-ghttm"] Sep 15 11:37:28.261167 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.260966 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.263599 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.263583 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 15 11:37:28.263698 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.263583 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 15 11:37:28.264188 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.264162 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 15 11:37:28.264322 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.264254 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 15 11:37:28.264650 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.264624 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-sdhx4\"" Sep 15 11:37:28.268774 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.268752 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 15 11:37:28.280562 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.280540 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j"] Sep 15 11:37:28.280703 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.280677 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.283116 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.283095 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 15 11:37:28.283116 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.283114 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 15 11:37:28.283422 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.283407 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pvn7w\"" Sep 15 11:37:28.283492 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.283438 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 15 11:37:28.283492 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.283485 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 15 11:37:28.283592 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.283438 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 15 11:37:28.283761 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.283743 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 15 11:37:28.302768 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.302743 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2lz98"] Sep 15 11:37:28.302912 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.302815 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" Sep 15 11:37:28.305512 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.305495 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 15 11:37:28.305626 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.305534 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 15 11:37:28.305626 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.305597 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-j5xw5\"" Sep 15 11:37:28.305626 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.305617 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 15 11:37:28.305785 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.305696 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 15 11:37:28.314805 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.314769 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd"] Sep 15 11:37:28.314908 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.314891 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2lz98" Sep 15 11:37:28.317396 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.317378 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 15 11:37:28.317715 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.317698 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9tznn\"" Sep 15 11:37:28.317906 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.317888 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 15 11:37:28.333296 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.333273 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.333404 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.333309 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed2b367e-e74c-4a65-92a7-ea170e652aae-installation-pull-secrets\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.333404 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.333347 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:28.333404 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.333371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-bound-sa-token\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.333560 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.333433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed2b367e-e74c-4a65-92a7-ea170e652aae-ca-trust-extracted\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.333560 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.333455 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed2b367e-e74c-4a65-92a7-ea170e652aae-trusted-ca\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.333560 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.333529 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-certificates\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.333702 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.333566 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbf9f\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-kube-api-access-qbf9f\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.333702 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.333586 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bbdaac6c-adf2-43e0-ae77-954292189938-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:28.333702 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.333646 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxgsv\" (UniqueName: \"kubernetes.io/projected/bbdaac6c-adf2-43e0-ae77-954292189938-kube-api-access-zxgsv\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:28.333844 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.333713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed2b367e-e74c-4a65-92a7-ea170e652aae-image-registry-private-configuration\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.334976 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.334955 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-nzs6t"] Sep 15 11:37:28.335131 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.335114 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:37:28.337772 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.337615 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 15 11:37:28.337772 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.337677 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 15 11:37:28.337936 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.337796 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zcsmb\"" Sep 15 11:37:28.338098 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.338081 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 15 11:37:28.354841 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.354822 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5"] Sep 15 11:37:28.354934 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.354855 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2lz98"] Sep 15 11:37:28.354934 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.354875 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cf4d4794-zt4c6"] Sep 15 11:37:28.354934 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.354897 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hp6hw"] Sep 15 11:37:28.354934 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.354929 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.357572 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.357551 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 15 11:37:28.357680 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.357643 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 15 11:37:28.357738 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.357694 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 15 11:37:28.357738 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.357699 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 15 11:37:28.357812 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.357757 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lzx5t\"" Sep 15 11:37:28.366514 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.366491 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 15 11:37:28.379239 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.379219 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-hd9fz"] Sep 15 11:37:28.379308 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.379246 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz"] Sep 15 11:37:28.379352 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.379330 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hp6hw" Sep 15 11:37:28.381959 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.381943 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 15 11:37:28.382093 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.382077 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t7jqq\"" Sep 15 11:37:28.382174 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.382150 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 15 11:37:28.382174 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.382150 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 15 11:37:28.401219 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.401201 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-nzs6t"] Sep 15 11:37:28.401302 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.401228 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5ddcf9f498-ghttm"] Sep 15 11:37:28.401302 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.401242 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-94pfg"] Sep 15 11:37:28.401374 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.401318 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" Sep 15 11:37:28.403800 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.403777 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 15 11:37:28.403800 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.403787 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 15 11:37:28.403953 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.403830 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 15 11:37:28.403953 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.403842 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m6wk6\"" Sep 15 11:37:28.403953 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.403850 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 15 11:37:28.419489 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.419448 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-6nqxb"] Sep 15 11:37:28.419587 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.419572 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:28.422062 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.422043 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 15 11:37:28.422160 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.422049 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qkkmw\"" Sep 15 11:37:28.422160 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.422048 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 15 11:37:28.434952 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.434933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed2b367e-e74c-4a65-92a7-ea170e652aae-installation-pull-secrets\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.435050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.434961 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj75m\" (UniqueName: \"kubernetes.io/projected/22eeb9ed-5ded-442e-82ae-ac31f979e681-kube-api-access-zj75m\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.435050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.434978 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.435050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.434995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-default-certificate\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.435174 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435141 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:28.435174 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-bound-sa-token\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.435240 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435186 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22eeb9ed-5ded-442e-82ae-ac31f979e681-tmp\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.435240 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/22eeb9ed-5ded-442e-82ae-ac31f979e681-snapshots\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.435337 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.435246 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 11:37:28.435337 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435247 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96nmr\" (UniqueName: \"kubernetes.io/projected/49259c18-d5c4-4a14-b715-60dae1b8a2ef-kube-api-access-96nmr\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6f89j\" (UID: \"49259c18-d5c4-4a14-b715-60dae1b8a2ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" Sep 15 11:37:28.435337 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435302 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed2b367e-e74c-4a65-92a7-ea170e652aae-ca-trust-extracted\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.435439 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.435352 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls podName:bbdaac6c-adf2-43e0-ae77-954292189938 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:28.935333016 +0000 UTC m=+36.165259534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-n6tc5" (UID: "bbdaac6c-adf2-43e0-ae77-954292189938") : secret "cluster-monitoring-operator-tls" not found Sep 15 11:37:28.435439 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed2b367e-e74c-4a65-92a7-ea170e652aae-trusted-ca\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.435558 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435453 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-certificates\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.435558 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435480 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbf9f\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-kube-api-access-qbf9f\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.435558 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435506 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bbdaac6c-adf2-43e0-ae77-954292189938-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:28.435558 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435538 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxgsv\" (UniqueName: \"kubernetes.io/projected/bbdaac6c-adf2-43e0-ae77-954292189938-kube-api-access-zxgsv\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:28.435743 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435566 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22eeb9ed-5ded-442e-82ae-ac31f979e681-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.435743 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22eeb9ed-5ded-442e-82ae-ac31f979e681-service-ca-bundle\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.435743 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435635 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49259c18-d5c4-4a14-b715-60dae1b8a2ef-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6f89j\" (UID: \"49259c18-d5c4-4a14-b715-60dae1b8a2ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" Sep 15 11:37:28.435743 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435659 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49259c18-d5c4-4a14-b715-60dae1b8a2ef-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6f89j\" (UID: \"49259c18-d5c4-4a14-b715-60dae1b8a2ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" Sep 15 11:37:28.435743 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxq5s\" (UniqueName: \"kubernetes.io/projected/278b23ea-e4b2-440d-8c62-2cea30acb706-kube-api-access-vxq5s\") pod \"volume-data-source-validator-86695b7cc7-2lz98\" (UID: \"278b23ea-e4b2-440d-8c62-2cea30acb706\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2lz98" Sep 15 11:37:28.435743 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435694 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed2b367e-e74c-4a65-92a7-ea170e652aae-ca-trust-extracted\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.435743 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22eeb9ed-5ded-442e-82ae-ac31f979e681-serving-cert\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.435743 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435739 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h8jv\" (UniqueName: \"kubernetes.io/projected/78d07596-2598-4c06-a63c-ecb48c2052a0-kube-api-access-6h8jv\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.436150 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435810 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed2b367e-e74c-4a65-92a7-ea170e652aae-image-registry-private-configuration\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.436150 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435847 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-stats-auth\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.436150 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435914 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7tnr\" (UniqueName: \"kubernetes.io/projected/a247eace-c065-4b91-b2e5-a2d0c88cae08-kube-api-access-n7tnr\") pod \"cluster-samples-operator-86499c89c4-9slkd\" (UID: \"a247eace-c065-4b91-b2e5-a2d0c88cae08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:37:28.436150 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435942 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9slkd\" (UID: \"a247eace-c065-4b91-b2e5-a2d0c88cae08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:37:28.436150 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435971 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.436150 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.435997 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.436150 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.436051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-certificates\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.436150 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.436083 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 11:37:28.436150 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.436096 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4d4794-zt4c6: secret "image-registry-tls" not found Sep 15 11:37:28.436150 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.436135 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls podName:ed2b367e-e74c-4a65-92a7-ea170e652aae nodeName:}" failed. No retries permitted until 2026-09-15 11:37:28.936122483 +0000 UTC m=+36.166049002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls") pod "image-registry-7cf4d4794-zt4c6" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae") : secret "image-registry-tls" not found Sep 15 11:37:28.436602 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.436374 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bbdaac6c-adf2-43e0-ae77-954292189938-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:28.436602 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.436490 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed2b367e-e74c-4a65-92a7-ea170e652aae-trusted-ca\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.439252 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.439228 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed2b367e-e74c-4a65-92a7-ea170e652aae-installation-pull-secrets\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.439350 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.439270 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed2b367e-e74c-4a65-92a7-ea170e652aae-image-registry-private-configuration\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.447576 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.447552 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbf9f\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-kube-api-access-qbf9f\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.447681 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.447552 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-bound-sa-token\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.448622 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.448598 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxgsv\" (UniqueName: \"kubernetes.io/projected/bbdaac6c-adf2-43e0-ae77-954292189938-kube-api-access-zxgsv\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:28.451712 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.451691 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j"] Sep 15 11:37:28.451816 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.451728 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz"] Sep 15 11:37:28.451816 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.451750 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hp6hw"] Sep 15 11:37:28.451816 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.451773 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd"] Sep 15 11:37:28.451816 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.451798 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94pfg"] Sep 15 11:37:28.451816 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.451804 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6nqxb" Sep 15 11:37:28.452091 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.451827 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-6nqxb"] Sep 15 11:37:28.454357 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.454324 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vc88p\"" Sep 15 11:37:28.536424 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/22eeb9ed-5ded-442e-82ae-ac31f979e681-snapshots\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.536603 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536441 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a645fa8f-dad2-486b-b3af-7ee707deaa93-cert\") pod \"ingress-canary-hp6hw\" (UID: \"a645fa8f-dad2-486b-b3af-7ee707deaa93\") " pod="openshift-ingress-canary/ingress-canary-hp6hw" Sep 15 11:37:28.536603 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536566 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/590c2c95-948f-4758-8c73-c327a005adf1-config\") pod \"console-operator-696884fb9c-nzs6t\" (UID: \"590c2c95-948f-4758-8c73-c327a005adf1\") " pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.536700 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536605 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76f77d07-00dd-441c-82ab-d037c4a6da6c-serving-cert\") pod \"service-ca-operator-cf7c776b7-4cfrz\" (UID: \"76f77d07-00dd-441c-82ab-d037c4a6da6c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" Sep 15 11:37:28.536700 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536671 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49259c18-d5c4-4a14-b715-60dae1b8a2ef-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6f89j\" (UID: \"49259c18-d5c4-4a14-b715-60dae1b8a2ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" Sep 15 11:37:28.536777 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536704 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxq5s\" (UniqueName: \"kubernetes.io/projected/278b23ea-e4b2-440d-8c62-2cea30acb706-kube-api-access-vxq5s\") pod \"volume-data-source-validator-86695b7cc7-2lz98\" (UID: \"278b23ea-e4b2-440d-8c62-2cea30acb706\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2lz98" Sep 15 11:37:28.536777 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:28.536777 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536761 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76f77d07-00dd-441c-82ab-d037c4a6da6c-config\") pod \"service-ca-operator-cf7c776b7-4cfrz\" (UID: \"76f77d07-00dd-441c-82ab-d037c4a6da6c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" Sep 15 11:37:28.536924 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536787 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-stats-auth\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.536924 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7tnr\" (UniqueName: \"kubernetes.io/projected/a247eace-c065-4b91-b2e5-a2d0c88cae08-kube-api-access-n7tnr\") pod \"cluster-samples-operator-86499c89c4-9slkd\" (UID: \"a247eace-c065-4b91-b2e5-a2d0c88cae08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:37:28.536924 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr4lj\" (UniqueName: \"kubernetes.io/projected/f051f18a-7e20-4574-91c1-2a8d9b3fe6cb-kube-api-access-zr4lj\") pod \"network-check-source-6f8b768455-6nqxb\" (UID: \"f051f18a-7e20-4574-91c1-2a8d9b3fe6cb\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-6nqxb" Sep 15 11:37:28.536924 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536904 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/590c2c95-948f-4758-8c73-c327a005adf1-serving-cert\") pod \"console-operator-696884fb9c-nzs6t\" (UID: \"590c2c95-948f-4758-8c73-c327a005adf1\") " pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.537140 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536939 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78499\" (UniqueName: \"kubernetes.io/projected/76f77d07-00dd-441c-82ab-d037c4a6da6c-kube-api-access-78499\") pod \"service-ca-operator-cf7c776b7-4cfrz\" (UID: \"76f77d07-00dd-441c-82ab-d037c4a6da6c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" Sep 15 11:37:28.537140 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.536975 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj75m\" (UniqueName: \"kubernetes.io/projected/22eeb9ed-5ded-442e-82ae-ac31f979e681-kube-api-access-zj75m\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.537140 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.537140 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537086 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96nmr\" (UniqueName: \"kubernetes.io/projected/49259c18-d5c4-4a14-b715-60dae1b8a2ef-kube-api-access-96nmr\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6f89j\" (UID: \"49259c18-d5c4-4a14-b715-60dae1b8a2ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" Sep 15 11:37:28.537140 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537122 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22eeb9ed-5ded-442e-82ae-ac31f979e681-service-ca-bundle\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.537375 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537150 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9slkd\" (UID: \"a247eace-c065-4b91-b2e5-a2d0c88cae08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:37:28.537375 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22eeb9ed-5ded-442e-82ae-ac31f979e681-serving-cert\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.537375 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537199 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/22eeb9ed-5ded-442e-82ae-ac31f979e681-snapshots\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.537375 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537206 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49259c18-d5c4-4a14-b715-60dae1b8a2ef-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6f89j\" (UID: \"49259c18-d5c4-4a14-b715-60dae1b8a2ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" Sep 15 11:37:28.537375 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22eeb9ed-5ded-442e-82ae-ac31f979e681-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.537375 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537283 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49259c18-d5c4-4a14-b715-60dae1b8a2ef-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6f89j\" (UID: \"49259c18-d5c4-4a14-b715-60dae1b8a2ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" Sep 15 11:37:28.537375 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537300 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvdv\" (UniqueName: \"kubernetes.io/projected/590c2c95-948f-4758-8c73-c327a005adf1-kube-api-access-jcvdv\") pod \"console-operator-696884fb9c-nzs6t\" (UID: \"590c2c95-948f-4758-8c73-c327a005adf1\") " pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.537375 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537363 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/590c2c95-948f-4758-8c73-c327a005adf1-trusted-ca\") pod \"console-operator-696884fb9c-nzs6t\" (UID: \"590c2c95-948f-4758-8c73-c327a005adf1\") " pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.537764 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h8jv\" (UniqueName: \"kubernetes.io/projected/78d07596-2598-4c06-a63c-ecb48c2052a0-kube-api-access-6h8jv\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.537764 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.537585 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle podName:78d07596-2598-4c06-a63c-ecb48c2052a0 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:29.037563339 +0000 UTC m=+36.267489857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle") pod "router-default-5ddcf9f498-ghttm" (UID: "78d07596-2598-4c06-a63c-ecb48c2052a0") : configmap references non-existent config key: service-ca.crt Sep 15 11:37:28.537764 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537610 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ddf\" (UniqueName: \"kubernetes.io/projected/39b7b0de-0d7f-4438-bd33-6a189e691139-kube-api-access-p9ddf\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:28.537764 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.537764 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537698 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/39b7b0de-0d7f-4438-bd33-6a189e691139-config-volume\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:28.537764 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537722 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b7b0de-0d7f-4438-bd33-6a189e691139-tmp-dir\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:28.537764 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.537754 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-default-certificate\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.538141 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.537812 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 11:37:28.538141 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.537848 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 11:37:28.538239 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.538140 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls podName:a247eace-c065-4b91-b2e5-a2d0c88cae08 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:29.038123555 +0000 UTC m=+36.268050068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9slkd" (UID: "a247eace-c065-4b91-b2e5-a2d0c88cae08") : secret "samples-operator-tls" not found Sep 15 11:37:28.538239 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.538176 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22eeb9ed-5ded-442e-82ae-ac31f979e681-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.538239 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.538186 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs podName:78d07596-2598-4c06-a63c-ecb48c2052a0 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:29.038176886 +0000 UTC m=+36.268103404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs") pod "router-default-5ddcf9f498-ghttm" (UID: "78d07596-2598-4c06-a63c-ecb48c2052a0") : secret "router-metrics-certs-default" not found Sep 15 11:37:28.538398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.538014 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22eeb9ed-5ded-442e-82ae-ac31f979e681-tmp\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.538398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.538292 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v759\" (UniqueName: \"kubernetes.io/projected/a645fa8f-dad2-486b-b3af-7ee707deaa93-kube-api-access-9v759\") pod \"ingress-canary-hp6hw\" (UID: \"a645fa8f-dad2-486b-b3af-7ee707deaa93\") " pod="openshift-ingress-canary/ingress-canary-hp6hw" Sep 15 11:37:28.538398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.538338 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22eeb9ed-5ded-442e-82ae-ac31f979e681-service-ca-bundle\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.538546 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.538522 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22eeb9ed-5ded-442e-82ae-ac31f979e681-tmp\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.539781 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.539758 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49259c18-d5c4-4a14-b715-60dae1b8a2ef-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6f89j\" (UID: \"49259c18-d5c4-4a14-b715-60dae1b8a2ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" Sep 15 11:37:28.539930 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.539909 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22eeb9ed-5ded-442e-82ae-ac31f979e681-serving-cert\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.539993 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.539980 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-stats-auth\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.540474 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.540453 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-default-certificate\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.549554 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.549529 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7tnr\" (UniqueName: \"kubernetes.io/projected/a247eace-c065-4b91-b2e5-a2d0c88cae08-kube-api-access-n7tnr\") pod \"cluster-samples-operator-86499c89c4-9slkd\" (UID: \"a247eace-c065-4b91-b2e5-a2d0c88cae08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:37:28.550657 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.550631 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h8jv\" (UniqueName: \"kubernetes.io/projected/78d07596-2598-4c06-a63c-ecb48c2052a0-kube-api-access-6h8jv\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:28.550861 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.550842 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96nmr\" (UniqueName: \"kubernetes.io/projected/49259c18-d5c4-4a14-b715-60dae1b8a2ef-kube-api-access-96nmr\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6f89j\" (UID: \"49259c18-d5c4-4a14-b715-60dae1b8a2ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" Sep 15 11:37:28.550941 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.550909 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj75m\" (UniqueName: \"kubernetes.io/projected/22eeb9ed-5ded-442e-82ae-ac31f979e681-kube-api-access-zj75m\") pod \"insights-operator-5844b856db-hd9fz\" (UID: \"22eeb9ed-5ded-442e-82ae-ac31f979e681\") " pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.551087 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.550959 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxq5s\" (UniqueName: \"kubernetes.io/projected/278b23ea-e4b2-440d-8c62-2cea30acb706-kube-api-access-vxq5s\") pod \"volume-data-source-validator-86695b7cc7-2lz98\" (UID: \"278b23ea-e4b2-440d-8c62-2cea30acb706\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2lz98" Sep 15 11:37:28.584662 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.584635 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-hd9fz" Sep 15 11:37:28.612091 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.612066 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" Sep 15 11:37:28.624624 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.624596 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2lz98" Sep 15 11:37:28.639451 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639425 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ddf\" (UniqueName: \"kubernetes.io/projected/39b7b0de-0d7f-4438-bd33-6a189e691139-kube-api-access-p9ddf\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:28.639551 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/39b7b0de-0d7f-4438-bd33-6a189e691139-config-volume\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:28.639551 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b7b0de-0d7f-4438-bd33-6a189e691139-tmp-dir\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:28.639669 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v759\" (UniqueName: \"kubernetes.io/projected/a645fa8f-dad2-486b-b3af-7ee707deaa93-kube-api-access-9v759\") pod \"ingress-canary-hp6hw\" (UID: \"a645fa8f-dad2-486b-b3af-7ee707deaa93\") " pod="openshift-ingress-canary/ingress-canary-hp6hw" Sep 15 11:37:28.639669 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639595 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a645fa8f-dad2-486b-b3af-7ee707deaa93-cert\") pod \"ingress-canary-hp6hw\" (UID: \"a645fa8f-dad2-486b-b3af-7ee707deaa93\") " pod="openshift-ingress-canary/ingress-canary-hp6hw" Sep 15 11:37:28.639669 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/590c2c95-948f-4758-8c73-c327a005adf1-config\") pod \"console-operator-696884fb9c-nzs6t\" (UID: \"590c2c95-948f-4758-8c73-c327a005adf1\") " pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.639669 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76f77d07-00dd-441c-82ab-d037c4a6da6c-serving-cert\") pod \"service-ca-operator-cf7c776b7-4cfrz\" (UID: \"76f77d07-00dd-441c-82ab-d037c4a6da6c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" Sep 15 11:37:28.639862 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:28.639862 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76f77d07-00dd-441c-82ab-d037c4a6da6c-config\") pod \"service-ca-operator-cf7c776b7-4cfrz\" (UID: \"76f77d07-00dd-441c-82ab-d037c4a6da6c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" Sep 15 11:37:28.639862 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr4lj\" (UniqueName: \"kubernetes.io/projected/f051f18a-7e20-4574-91c1-2a8d9b3fe6cb-kube-api-access-zr4lj\") pod \"network-check-source-6f8b768455-6nqxb\" (UID: \"f051f18a-7e20-4574-91c1-2a8d9b3fe6cb\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-6nqxb" Sep 15 11:37:28.639862 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/590c2c95-948f-4758-8c73-c327a005adf1-serving-cert\") pod \"console-operator-696884fb9c-nzs6t\" (UID: \"590c2c95-948f-4758-8c73-c327a005adf1\") " pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.639862 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78499\" (UniqueName: \"kubernetes.io/projected/76f77d07-00dd-441c-82ab-d037c4a6da6c-kube-api-access-78499\") pod \"service-ca-operator-cf7c776b7-4cfrz\" (UID: \"76f77d07-00dd-441c-82ab-d037c4a6da6c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" Sep 15 11:37:28.640130 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639881 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b7b0de-0d7f-4438-bd33-6a189e691139-tmp-dir\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:28.640130 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.639975 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvdv\" (UniqueName: \"kubernetes.io/projected/590c2c95-948f-4758-8c73-c327a005adf1-kube-api-access-jcvdv\") pod \"console-operator-696884fb9c-nzs6t\" (UID: \"590c2c95-948f-4758-8c73-c327a005adf1\") " pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.640130 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.640016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/590c2c95-948f-4758-8c73-c327a005adf1-trusted-ca\") pod \"console-operator-696884fb9c-nzs6t\" (UID: \"590c2c95-948f-4758-8c73-c327a005adf1\") " pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.640269 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.640182 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/39b7b0de-0d7f-4438-bd33-6a189e691139-config-volume\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:28.640902 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.640657 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/590c2c95-948f-4758-8c73-c327a005adf1-config\") pod \"console-operator-696884fb9c-nzs6t\" (UID: \"590c2c95-948f-4758-8c73-c327a005adf1\") " pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.640902 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.640276 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 11:37:28.640902 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.640732 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls podName:39b7b0de-0d7f-4438-bd33-6a189e691139 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:29.140712732 +0000 UTC m=+36.370639260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls") pod "dns-default-94pfg" (UID: "39b7b0de-0d7f-4438-bd33-6a189e691139") : secret "dns-default-metrics-tls" not found Sep 15 11:37:28.640902 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.640831 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76f77d07-00dd-441c-82ab-d037c4a6da6c-config\") pod \"service-ca-operator-cf7c776b7-4cfrz\" (UID: \"76f77d07-00dd-441c-82ab-d037c4a6da6c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" Sep 15 11:37:28.641210 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.640916 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/590c2c95-948f-4758-8c73-c327a005adf1-trusted-ca\") pod \"console-operator-696884fb9c-nzs6t\" (UID: \"590c2c95-948f-4758-8c73-c327a005adf1\") " pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.642818 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.642794 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76f77d07-00dd-441c-82ab-d037c4a6da6c-serving-cert\") pod \"service-ca-operator-cf7c776b7-4cfrz\" (UID: \"76f77d07-00dd-441c-82ab-d037c4a6da6c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" Sep 15 11:37:28.642956 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.642929 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/590c2c95-948f-4758-8c73-c327a005adf1-serving-cert\") pod \"console-operator-696884fb9c-nzs6t\" (UID: \"590c2c95-948f-4758-8c73-c327a005adf1\") " pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.643071 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.643005 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a645fa8f-dad2-486b-b3af-7ee707deaa93-cert\") pod \"ingress-canary-hp6hw\" (UID: \"a645fa8f-dad2-486b-b3af-7ee707deaa93\") " pod="openshift-ingress-canary/ingress-canary-hp6hw" Sep 15 11:37:28.649771 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.649742 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78499\" (UniqueName: \"kubernetes.io/projected/76f77d07-00dd-441c-82ab-d037c4a6da6c-kube-api-access-78499\") pod \"service-ca-operator-cf7c776b7-4cfrz\" (UID: \"76f77d07-00dd-441c-82ab-d037c4a6da6c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" Sep 15 11:37:28.649883 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.649748 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v759\" (UniqueName: \"kubernetes.io/projected/a645fa8f-dad2-486b-b3af-7ee707deaa93-kube-api-access-9v759\") pod \"ingress-canary-hp6hw\" (UID: \"a645fa8f-dad2-486b-b3af-7ee707deaa93\") " pod="openshift-ingress-canary/ingress-canary-hp6hw" Sep 15 11:37:28.649950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.649881 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvdv\" (UniqueName: \"kubernetes.io/projected/590c2c95-948f-4758-8c73-c327a005adf1-kube-api-access-jcvdv\") pod \"console-operator-696884fb9c-nzs6t\" (UID: \"590c2c95-948f-4758-8c73-c327a005adf1\") " pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.649950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.649917 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ddf\" (UniqueName: \"kubernetes.io/projected/39b7b0de-0d7f-4438-bd33-6a189e691139-kube-api-access-p9ddf\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:28.650404 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.650387 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr4lj\" (UniqueName: \"kubernetes.io/projected/f051f18a-7e20-4574-91c1-2a8d9b3fe6cb-kube-api-access-zr4lj\") pod \"network-check-source-6f8b768455-6nqxb\" (UID: \"f051f18a-7e20-4574-91c1-2a8d9b3fe6cb\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-6nqxb" Sep 15 11:37:28.663465 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.663440 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:28.686987 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.686962 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hp6hw" Sep 15 11:37:28.708836 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.708818 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" Sep 15 11:37:28.763326 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.763295 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6nqxb" Sep 15 11:37:28.943641 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.943534 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:28.943817 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:28.943658 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:28.943817 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.943779 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 11:37:28.943933 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.943840 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 11:37:28.943933 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.943862 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4d4794-zt4c6: secret "image-registry-tls" not found Sep 15 11:37:28.943933 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.943862 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls podName:bbdaac6c-adf2-43e0-ae77-954292189938 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:29.943839526 +0000 UTC m=+37.173766041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-n6tc5" (UID: "bbdaac6c-adf2-43e0-ae77-954292189938") : secret "cluster-monitoring-operator-tls" not found Sep 15 11:37:28.943933 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:28.943929 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls podName:ed2b367e-e74c-4a65-92a7-ea170e652aae nodeName:}" failed. No retries permitted until 2026-09-15 11:37:29.943911722 +0000 UTC m=+37.173838237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls") pod "image-registry-7cf4d4794-zt4c6" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae") : secret "image-registry-tls" not found Sep 15 11:37:29.044350 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:29.044315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9slkd\" (UID: \"a247eace-c065-4b91-b2e5-a2d0c88cae08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:37:29.044518 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:29.044405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:29.044518 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:29.044493 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 11:37:29.044634 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:29.044569 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls podName:a247eace-c065-4b91-b2e5-a2d0c88cae08 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:30.044550033 +0000 UTC m=+37.274476551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9slkd" (UID: "a247eace-c065-4b91-b2e5-a2d0c88cae08") : secret "samples-operator-tls" not found Sep 15 11:37:29.044634 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:29.044601 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle podName:78d07596-2598-4c06-a63c-ecb48c2052a0 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:30.044584674 +0000 UTC m=+37.274511204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle") pod "router-default-5ddcf9f498-ghttm" (UID: "78d07596-2598-4c06-a63c-ecb48c2052a0") : configmap references non-existent config key: service-ca.crt Sep 15 11:37:29.044757 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:29.044672 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 11:37:29.044757 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:29.044709 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs podName:78d07596-2598-4c06-a63c-ecb48c2052a0 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:30.044698166 +0000 UTC m=+37.274624698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs") pod "router-default-5ddcf9f498-ghttm" (UID: "78d07596-2598-4c06-a63c-ecb48c2052a0") : secret "router-metrics-certs-default" not found Sep 15 11:37:29.044757 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:29.044502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:29.145464 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:29.145426 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:29.145643 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:29.145556 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 11:37:29.145643 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:29.145629 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls podName:39b7b0de-0d7f-4438-bd33-6a189e691139 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:30.145606306 +0000 UTC m=+37.375532838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls") pod "dns-default-94pfg" (UID: "39b7b0de-0d7f-4438-bd33-6a189e691139") : secret "dns-default-metrics-tls" not found Sep 15 11:37:29.229328 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:29.229232 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:29.232349 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:29.232258 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 11:37:29.952613 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:29.952578 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:29.952780 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:29.952689 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:29.952780 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:29.952768 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 11:37:29.952888 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:29.952832 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls podName:bbdaac6c-adf2-43e0-ae77-954292189938 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:31.952815185 +0000 UTC m=+39.182741705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-n6tc5" (UID: "bbdaac6c-adf2-43e0-ae77-954292189938") : secret "cluster-monitoring-operator-tls" not found Sep 15 11:37:29.952888 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:29.952867 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 11:37:29.952990 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:29.952892 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4d4794-zt4c6: secret "image-registry-tls" not found Sep 15 11:37:29.952990 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:29.952958 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls podName:ed2b367e-e74c-4a65-92a7-ea170e652aae nodeName:}" failed. No retries permitted until 2026-09-15 11:37:31.952936412 +0000 UTC m=+39.182862938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls") pod "image-registry-7cf4d4794-zt4c6" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae") : secret "image-registry-tls" not found Sep 15 11:37:30.055839 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.054697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:30.055839 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.054794 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9slkd\" (UID: \"a247eace-c065-4b91-b2e5-a2d0c88cae08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:37:30.055839 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.054883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:30.055839 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:30.055100 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 11:37:30.055839 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:30.055173 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs podName:78d07596-2598-4c06-a63c-ecb48c2052a0 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:32.055152789 +0000 UTC m=+39.285079317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs") pod "router-default-5ddcf9f498-ghttm" (UID: "78d07596-2598-4c06-a63c-ecb48c2052a0") : secret "router-metrics-certs-default" not found Sep 15 11:37:30.055839 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:30.055672 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle podName:78d07596-2598-4c06-a63c-ecb48c2052a0 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:32.055653386 +0000 UTC m=+39.285579914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle") pod "router-default-5ddcf9f498-ghttm" (UID: "78d07596-2598-4c06-a63c-ecb48c2052a0") : configmap references non-existent config key: service-ca.crt Sep 15 11:37:30.055839 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:30.055755 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 11:37:30.055839 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:30.055794 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls podName:a247eace-c065-4b91-b2e5-a2d0c88cae08 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:32.055780394 +0000 UTC m=+39.285706923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9slkd" (UID: "a247eace-c065-4b91-b2e5-a2d0c88cae08") : secret "samples-operator-tls" not found Sep 15 11:37:30.155834 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.155801 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:30.155979 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:30.155961 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 11:37:30.156072 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:30.156060 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls podName:39b7b0de-0d7f-4438-bd33-6a189e691139 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:32.156010544 +0000 UTC m=+39.385937072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls") pod "dns-default-94pfg" (UID: "39b7b0de-0d7f-4438-bd33-6a189e691139") : secret "dns-default-metrics-tls" not found Sep 15 11:37:30.320138 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.320107 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-hd9fz"] Sep 15 11:37:30.325905 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:37:30.325878 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22eeb9ed_5ded_442e_82ae_ac31f979e681.slice/crio-505106bd57eb96f481e39d6741244767b5ff8e7b028cff08d01741cd43950465 WatchSource:0}: Error finding container 505106bd57eb96f481e39d6741244767b5ff8e7b028cff08d01741cd43950465: Status 404 returned error can't find the container with id 505106bd57eb96f481e39d6741244767b5ff8e7b028cff08d01741cd43950465 Sep 15 11:37:30.332177 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.332144 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz"] Sep 15 11:37:30.335759 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.335716 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-6nqxb"] Sep 15 11:37:30.337933 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:37:30.337379 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f77d07_00dd_441c_82ab_d037c4a6da6c.slice/crio-08283712d3904bbddeb292600983bc52a96201ad6a634ca761895cf12fc74ba5 WatchSource:0}: Error finding container 08283712d3904bbddeb292600983bc52a96201ad6a634ca761895cf12fc74ba5: Status 404 returned error can't find the container with id 08283712d3904bbddeb292600983bc52a96201ad6a634ca761895cf12fc74ba5 Sep 15 11:37:30.338751 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.338730 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-nzs6t"] Sep 15 11:37:30.340148 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:37:30.340015 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590c2c95_948f_4758_8c73_c327a005adf1.slice/crio-ed88947d0c184d22bf5290c80760d34998f3631f1f2bf0a8c9366f1baffa0003 WatchSource:0}: Error finding container ed88947d0c184d22bf5290c80760d34998f3631f1f2bf0a8c9366f1baffa0003: Status 404 returned error can't find the container with id ed88947d0c184d22bf5290c80760d34998f3631f1f2bf0a8c9366f1baffa0003 Sep 15 11:37:30.340148 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.340068 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hp6hw"] Sep 15 11:37:30.341209 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.341190 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2lz98"] Sep 15 11:37:30.342158 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:37:30.342139 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda645fa8f_dad2_486b_b3af_7ee707deaa93.slice/crio-d9c3fb0e81595b2da3a9207ec99d602706861e43efce28fc590ef4b0d029e475 WatchSource:0}: Error finding container d9c3fb0e81595b2da3a9207ec99d602706861e43efce28fc590ef4b0d029e475: Status 404 returned error can't find the container with id d9c3fb0e81595b2da3a9207ec99d602706861e43efce28fc590ef4b0d029e475 Sep 15 11:37:30.342382 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.342370 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j"] Sep 15 11:37:30.343340 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:37:30.343284 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf051f18a_7e20_4574_91c1_2a8d9b3fe6cb.slice/crio-255514e37f9612d6a04a87f628887a7980b1e568ca1ea7ec1c51bf83ba91a4d3 WatchSource:0}: Error finding container 255514e37f9612d6a04a87f628887a7980b1e568ca1ea7ec1c51bf83ba91a4d3: Status 404 returned error can't find the container with id 255514e37f9612d6a04a87f628887a7980b1e568ca1ea7ec1c51bf83ba91a4d3 Sep 15 11:37:30.344354 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:37:30.344309 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49259c18_d5c4_4a14_b715_60dae1b8a2ef.slice/crio-482bc5ad16716ef567977155d8b4c309c7684241cc2341813c5159de95b879b2 WatchSource:0}: Error finding container 482bc5ad16716ef567977155d8b4c309c7684241cc2341813c5159de95b879b2: Status 404 returned error can't find the container with id 482bc5ad16716ef567977155d8b4c309c7684241cc2341813c5159de95b879b2 Sep 15 11:37:30.345153 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:37:30.345125 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278b23ea_e4b2_440d_8c62_2cea30acb706.slice/crio-4d2638ae6d1087da336ba8239afc770303055a535c88130ed184a17a74bd0cd3 WatchSource:0}: Error finding container 4d2638ae6d1087da336ba8239afc770303055a535c88130ed184a17a74bd0cd3: Status 404 returned error can't find the container with id 4d2638ae6d1087da336ba8239afc770303055a535c88130ed184a17a74bd0cd3 Sep 15 11:37:30.453424 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.453390 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" event={"ID":"590c2c95-948f-4758-8c73-c327a005adf1","Type":"ContainerStarted","Data":"ed88947d0c184d22bf5290c80760d34998f3631f1f2bf0a8c9366f1baffa0003"} Sep 15 11:37:30.454714 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.454689 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hd9fz" event={"ID":"22eeb9ed-5ded-442e-82ae-ac31f979e681","Type":"ContainerStarted","Data":"505106bd57eb96f481e39d6741244767b5ff8e7b028cff08d01741cd43950465"} Sep 15 11:37:30.455871 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.455843 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" event={"ID":"76f77d07-00dd-441c-82ab-d037c4a6da6c","Type":"ContainerStarted","Data":"08283712d3904bbddeb292600983bc52a96201ad6a634ca761895cf12fc74ba5"} Sep 15 11:37:30.456997 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.456974 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6nqxb" event={"ID":"f051f18a-7e20-4574-91c1-2a8d9b3fe6cb","Type":"ContainerStarted","Data":"255514e37f9612d6a04a87f628887a7980b1e568ca1ea7ec1c51bf83ba91a4d3"} Sep 15 11:37:30.458169 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.458141 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" event={"ID":"49259c18-d5c4-4a14-b715-60dae1b8a2ef","Type":"ContainerStarted","Data":"482bc5ad16716ef567977155d8b4c309c7684241cc2341813c5159de95b879b2"} Sep 15 11:37:30.459184 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.459165 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2lz98" event={"ID":"278b23ea-e4b2-440d-8c62-2cea30acb706","Type":"ContainerStarted","Data":"4d2638ae6d1087da336ba8239afc770303055a535c88130ed184a17a74bd0cd3"} Sep 15 11:37:30.460288 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.460266 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hp6hw" event={"ID":"a645fa8f-dad2-486b-b3af-7ee707deaa93","Type":"ContainerStarted","Data":"d9c3fb0e81595b2da3a9207ec99d602706861e43efce28fc590ef4b0d029e475"} Sep 15 11:37:30.463172 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:30.463152 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4mzd" event={"ID":"4b6f665e-7431-4599-b9ef-302eb545b1c9","Type":"ContainerStarted","Data":"23b7abaa861e49e900ec7834a0c94a21207ea88d3d4607f6c3c6a34165fb0f69"} Sep 15 11:37:31.477451 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:31.477327 2573 generic.go:356] "Generic (PLEG): container finished" podID="4b6f665e-7431-4599-b9ef-302eb545b1c9" containerID="23b7abaa861e49e900ec7834a0c94a21207ea88d3d4607f6c3c6a34165fb0f69" exitCode=0 Sep 15 11:37:31.477451 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:31.477376 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4mzd" event={"ID":"4b6f665e-7431-4599-b9ef-302eb545b1c9","Type":"ContainerDied","Data":"23b7abaa861e49e900ec7834a0c94a21207ea88d3d4607f6c3c6a34165fb0f69"} Sep 15 11:37:31.972036 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:31.971977 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:31.972213 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:31.972174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:31.972327 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:31.972308 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 11:37:31.972418 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:31.972394 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls podName:bbdaac6c-adf2-43e0-ae77-954292189938 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:35.972364664 +0000 UTC m=+43.202291186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-n6tc5" (UID: "bbdaac6c-adf2-43e0-ae77-954292189938") : secret "cluster-monitoring-operator-tls" not found Sep 15 11:37:31.972956 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:31.972844 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 11:37:31.972956 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:31.972863 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4d4794-zt4c6: secret "image-registry-tls" not found Sep 15 11:37:31.972956 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:31.972923 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls podName:ed2b367e-e74c-4a65-92a7-ea170e652aae nodeName:}" failed. No retries permitted until 2026-09-15 11:37:35.972895782 +0000 UTC m=+43.202822301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls") pod "image-registry-7cf4d4794-zt4c6" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae") : secret "image-registry-tls" not found Sep 15 11:37:32.074394 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:32.073224 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:32.074394 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:32.073407 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:32.074394 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:32.073465 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9slkd\" (UID: \"a247eace-c065-4b91-b2e5-a2d0c88cae08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:37:32.074394 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:32.073673 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 11:37:32.074394 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:32.073739 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls podName:a247eace-c065-4b91-b2e5-a2d0c88cae08 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:36.073719504 +0000 UTC m=+43.303646040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9slkd" (UID: "a247eace-c065-4b91-b2e5-a2d0c88cae08") : secret "samples-operator-tls" not found Sep 15 11:37:32.074394 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:32.074182 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 11:37:32.074394 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:32.074237 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs podName:78d07596-2598-4c06-a63c-ecb48c2052a0 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:36.074221673 +0000 UTC m=+43.304148203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs") pod "router-default-5ddcf9f498-ghttm" (UID: "78d07596-2598-4c06-a63c-ecb48c2052a0") : secret "router-metrics-certs-default" not found Sep 15 11:37:32.074394 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:32.074311 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle podName:78d07596-2598-4c06-a63c-ecb48c2052a0 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:36.074298513 +0000 UTC m=+43.304225051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle") pod "router-default-5ddcf9f498-ghttm" (UID: "78d07596-2598-4c06-a63c-ecb48c2052a0") : configmap references non-existent config key: service-ca.crt Sep 15 11:37:32.175893 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:32.174991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:32.175893 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:32.175456 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 11:37:32.175893 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:32.175521 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls podName:39b7b0de-0d7f-4438-bd33-6a189e691139 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:36.175501413 +0000 UTC m=+43.405427942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls") pod "dns-default-94pfg" (UID: "39b7b0de-0d7f-4438-bd33-6a189e691139") : secret "dns-default-metrics-tls" not found Sep 15 11:37:32.488043 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:32.487987 2573 generic.go:356] "Generic (PLEG): container finished" podID="4b6f665e-7431-4599-b9ef-302eb545b1c9" containerID="a9e7768e84424634b2c7ba9e0d57fce1b17f029ce1a58fa72b3bf9a1df15ff2f" exitCode=0 Sep 15 11:37:32.488465 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:32.488062 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4mzd" event={"ID":"4b6f665e-7431-4599-b9ef-302eb545b1c9","Type":"ContainerDied","Data":"a9e7768e84424634b2c7ba9e0d57fce1b17f029ce1a58fa72b3bf9a1df15ff2f"} Sep 15 11:37:35.374789 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:35.374720 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq"] Sep 15 11:37:35.405540 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:35.405489 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq"] Sep 15 11:37:35.405703 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:35.405617 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:37:35.411054 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:35.411011 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 15 11:37:35.411176 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:35.411066 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 15 11:37:35.411176 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:35.411069 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7vb4l\"" Sep 15 11:37:35.505604 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:35.505571 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q58lq\" (UID: \"dee2b36b-0dc3-4848-a413-ffca15516920\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:37:35.505781 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:35.505764 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dee2b36b-0dc3-4848-a413-ffca15516920-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-q58lq\" (UID: \"dee2b36b-0dc3-4848-a413-ffca15516920\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:37:35.606328 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:35.606294 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q58lq\" (UID: \"dee2b36b-0dc3-4848-a413-ffca15516920\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:37:35.606482 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:35.606441 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dee2b36b-0dc3-4848-a413-ffca15516920-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-q58lq\" (UID: \"dee2b36b-0dc3-4848-a413-ffca15516920\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:37:35.606482 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:35.606452 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 11:37:35.606596 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:35.606520 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert podName:dee2b36b-0dc3-4848-a413-ffca15516920 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:36.106498695 +0000 UTC m=+43.336425212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-q58lq" (UID: "dee2b36b-0dc3-4848-a413-ffca15516920") : secret "networking-console-plugin-cert" not found Sep 15 11:37:35.607066 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:35.607043 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dee2b36b-0dc3-4848-a413-ffca15516920-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-q58lq\" (UID: \"dee2b36b-0dc3-4848-a413-ffca15516920\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:37:36.010443 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:36.010397 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:36.010614 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:36.010511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:36.010614 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.010566 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 11:37:36.010737 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.010612 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 11:37:36.010737 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.010634 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4d4794-zt4c6: secret "image-registry-tls" not found Sep 15 11:37:36.010737 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.010659 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls podName:bbdaac6c-adf2-43e0-ae77-954292189938 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:44.010626304 +0000 UTC m=+51.240552841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-n6tc5" (UID: "bbdaac6c-adf2-43e0-ae77-954292189938") : secret "cluster-monitoring-operator-tls" not found Sep 15 11:37:36.010737 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.010692 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls podName:ed2b367e-e74c-4a65-92a7-ea170e652aae nodeName:}" failed. No retries permitted until 2026-09-15 11:37:44.010675178 +0000 UTC m=+51.240601717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls") pod "image-registry-7cf4d4794-zt4c6" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae") : secret "image-registry-tls" not found Sep 15 11:37:36.111653 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:36.111613 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:36.111841 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:36.111715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:36.111841 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:36.111740 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q58lq\" (UID: \"dee2b36b-0dc3-4848-a413-ffca15516920\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:37:36.111841 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:36.111774 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9slkd\" (UID: \"a247eace-c065-4b91-b2e5-a2d0c88cae08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:37:36.111841 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.111785 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 11:37:36.112057 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.111860 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs podName:78d07596-2598-4c06-a63c-ecb48c2052a0 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:44.11184024 +0000 UTC m=+51.341766754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs") pod "router-default-5ddcf9f498-ghttm" (UID: "78d07596-2598-4c06-a63c-ecb48c2052a0") : secret "router-metrics-certs-default" not found Sep 15 11:37:36.112057 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.111878 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 11:37:36.112057 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.111882 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 11:37:36.112057 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.111881 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle podName:78d07596-2598-4c06-a63c-ecb48c2052a0 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:44.111870604 +0000 UTC m=+51.341797123 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle") pod "router-default-5ddcf9f498-ghttm" (UID: "78d07596-2598-4c06-a63c-ecb48c2052a0") : configmap references non-existent config key: service-ca.crt Sep 15 11:37:36.112057 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.111961 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls podName:a247eace-c065-4b91-b2e5-a2d0c88cae08 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:44.111943225 +0000 UTC m=+51.341869755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9slkd" (UID: "a247eace-c065-4b91-b2e5-a2d0c88cae08") : secret "samples-operator-tls" not found Sep 15 11:37:36.112057 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.111976 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert podName:dee2b36b-0dc3-4848-a413-ffca15516920 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:37.11196711 +0000 UTC m=+44.341893631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-q58lq" (UID: "dee2b36b-0dc3-4848-a413-ffca15516920") : secret "networking-console-plugin-cert" not found Sep 15 11:37:36.212702 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:36.212664 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:36.212869 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.212778 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 11:37:36.212869 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:36.212853 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls podName:39b7b0de-0d7f-4438-bd33-6a189e691139 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:44.212837537 +0000 UTC m=+51.442764051 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls") pod "dns-default-94pfg" (UID: "39b7b0de-0d7f-4438-bd33-6a189e691139") : secret "dns-default-metrics-tls" not found Sep 15 11:37:37.121207 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:37.121171 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q58lq\" (UID: \"dee2b36b-0dc3-4848-a413-ffca15516920\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:37:37.121693 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:37.121288 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 11:37:37.121693 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:37.121355 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert podName:dee2b36b-0dc3-4848-a413-ffca15516920 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:39.121340229 +0000 UTC m=+46.351266744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-q58lq" (UID: "dee2b36b-0dc3-4848-a413-ffca15516920") : secret "networking-console-plugin-cert" not found Sep 15 11:37:38.507387 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.507345 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6nqxb" event={"ID":"f051f18a-7e20-4574-91c1-2a8d9b3fe6cb","Type":"ContainerStarted","Data":"950ef84f7a23a5b6f26c0ae146f7ca9bb915e67bf8e49dc04d1f06ec524c4618"} Sep 15 11:37:38.509385 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.509334 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" event={"ID":"49259c18-d5c4-4a14-b715-60dae1b8a2ef","Type":"ContainerStarted","Data":"f00e2c76b3b173ee673ac1742221132808270a30df3f756b51d2c0080db9825f"} Sep 15 11:37:38.511111 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.511054 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2lz98" event={"ID":"278b23ea-e4b2-440d-8c62-2cea30acb706","Type":"ContainerStarted","Data":"ef802bbc2d588f006dae7353da72432b02ecb7a05a39e5aeaef7f21dcd468839"} Sep 15 11:37:38.512791 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.512749 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hp6hw" event={"ID":"a645fa8f-dad2-486b-b3af-7ee707deaa93","Type":"ContainerStarted","Data":"4ca5f29fead7ba3becb85ad73f5c43fbe43094ab6a1535f4efc9569800201354"} Sep 15 11:37:38.516751 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.516729 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4mzd" event={"ID":"4b6f665e-7431-4599-b9ef-302eb545b1c9","Type":"ContainerStarted","Data":"cdceeffae7fb1cae719a0530f72f1e43e1d9a57b26b302c127615669e856e9d4"} Sep 15 11:37:38.518529 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.518509 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/0.log" Sep 15 11:37:38.518627 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.518549 2573 generic.go:356] "Generic (PLEG): container finished" podID="590c2c95-948f-4758-8c73-c327a005adf1" containerID="6bdfd64bef72f49a241afdee7147c0d725c46221eafad75544ce03bd4203edce" exitCode=255 Sep 15 11:37:38.518627 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.518616 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" event={"ID":"590c2c95-948f-4758-8c73-c327a005adf1","Type":"ContainerDied","Data":"6bdfd64bef72f49a241afdee7147c0d725c46221eafad75544ce03bd4203edce"} Sep 15 11:37:38.519006 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.518987 2573 scope.go:117] "RemoveContainer" containerID="6bdfd64bef72f49a241afdee7147c0d725c46221eafad75544ce03bd4203edce" Sep 15 11:37:38.520418 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.520398 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hd9fz" event={"ID":"22eeb9ed-5ded-442e-82ae-ac31f979e681","Type":"ContainerStarted","Data":"b8beeb5491b7b32939186e22ee736dfb22fcf4402a0f4e2e94f956d8d5b08cba"} Sep 15 11:37:38.522488 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.522467 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" event={"ID":"76f77d07-00dd-441c-82ab-d037c4a6da6c","Type":"ContainerStarted","Data":"f195f52bcdc8bf8b38f261435a153e60b8f44638e2be9ca3718ac1a574446d9f"} Sep 15 11:37:38.527820 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.527598 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6nqxb" podStartSLOduration=21.731402493 podStartE2EDuration="29.527585857s" totalImagesPullingTime="7.796183364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:38.527254225 +0000 UTC m=+45.757180764" watchObservedRunningTime="2026-09-15 11:37:38.527585857 +0000 UTC m=+45.757512385" Sep 15 11:37:38.546690 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.546636 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" podStartSLOduration=25.927361798 podStartE2EDuration="33.546618299s" totalImagesPullingTime="7.619256501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:38.545539974 +0000 UTC m=+45.775466512" watchObservedRunningTime="2026-09-15 11:37:38.546618299 +0000 UTC m=+45.776544836" Sep 15 11:37:38.563654 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.563609 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" podStartSLOduration=22.938215884999998 podStartE2EDuration="30.563590592s" totalImagesPullingTime="7.625374707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:38.563372289 +0000 UTC m=+45.793298829" watchObservedRunningTime="2026-09-15 11:37:38.563590592 +0000 UTC m=+45.793517120" Sep 15 11:37:38.585904 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.585824 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-hd9fz" podStartSLOduration=25.948655344 podStartE2EDuration="33.585804583s" totalImagesPullingTime="7.637149239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:38.584496326 +0000 UTC m=+45.814422856" watchObservedRunningTime="2026-09-15 11:37:38.585804583 +0000 UTC m=+45.815731116" Sep 15 11:37:38.603272 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.602792 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hp6hw" podStartSLOduration=2.971250219 podStartE2EDuration="10.602770594s" totalImagesPullingTime="7.631520375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:37:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:38.60236809 +0000 UTC m=+45.832294626" watchObservedRunningTime="2026-09-15 11:37:38.602770594 +0000 UTC m=+45.832697155" Sep 15 11:37:38.627452 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.625769 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m4mzd" podStartSLOduration=16.016190693 podStartE2EDuration="45.625748786s" totalImagesPullingTime="29.609558093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:36:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:38.625009918 +0000 UTC m=+45.854936444" watchObservedRunningTime="2026-09-15 11:37:38.625748786 +0000 UTC m=+45.855675323" Sep 15 11:37:38.664387 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.664357 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:38.664537 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.664417 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:38.671459 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:38.671193 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2lz98" podStartSLOduration=25.084660133 podStartE2EDuration="32.671174371s" totalImagesPullingTime="7.586514238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:38.650275175 +0000 UTC m=+45.880201709" watchObservedRunningTime="2026-09-15 11:37:38.671174371 +0000 UTC m=+45.901100908" Sep 15 11:37:39.141410 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.141362 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q58lq\" (UID: \"dee2b36b-0dc3-4848-a413-ffca15516920\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:37:39.141661 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:39.141549 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 11:37:39.141661 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:39.141638 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert podName:dee2b36b-0dc3-4848-a413-ffca15516920 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:43.14161602 +0000 UTC m=+50.371542540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-q58lq" (UID: "dee2b36b-0dc3-4848-a413-ffca15516920") : secret "networking-console-plugin-cert" not found Sep 15 11:37:39.531067 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.531038 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 11:37:39.531608 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.531580 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/0.log" Sep 15 11:37:39.531687 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.531624 2573 generic.go:356] "Generic (PLEG): container finished" podID="590c2c95-948f-4758-8c73-c327a005adf1" containerID="8931f8ab1c6333c73b9acff42a12088019dd324a2316f1438bfc4a343d92380c" exitCode=255 Sep 15 11:37:39.531738 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.531719 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" event={"ID":"590c2c95-948f-4758-8c73-c327a005adf1","Type":"ContainerDied","Data":"8931f8ab1c6333c73b9acff42a12088019dd324a2316f1438bfc4a343d92380c"} Sep 15 11:37:39.531791 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.531783 2573 scope.go:117] "RemoveContainer" containerID="6bdfd64bef72f49a241afdee7147c0d725c46221eafad75544ce03bd4203edce" Sep 15 11:37:39.532081 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.532059 2573 scope.go:117] "RemoveContainer" containerID="8931f8ab1c6333c73b9acff42a12088019dd324a2316f1438bfc4a343d92380c" Sep 15 11:37:39.532293 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:39.532268 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-nzs6t_openshift-console-operator(590c2c95-948f-4758-8c73-c327a005adf1)\"" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" podUID="590c2c95-948f-4758-8c73-c327a005adf1" Sep 15 11:37:39.792944 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.792865 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-599wc"] Sep 15 11:37:39.821878 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.821834 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-599wc"] Sep 15 11:37:39.822068 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.821959 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:39.824376 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.824352 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 15 11:37:39.824491 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.824476 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 15 11:37:39.825319 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.825298 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6nc5j\"" Sep 15 11:37:39.848467 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.848442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3351650b-9731-4e0b-a025-e65b6a1633ff-data-volume\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:39.848603 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.848577 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3351650b-9731-4e0b-a025-e65b6a1633ff-crio-socket\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:39.848673 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.848657 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:39.848711 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.848681 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:39.848752 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.848713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p4rt\" (UniqueName: \"kubernetes.io/projected/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-api-access-5p4rt\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:39.949403 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.949368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3351650b-9731-4e0b-a025-e65b6a1633ff-crio-socket\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:39.949597 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.949449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:39.949597 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.949477 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:39.949597 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.949491 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3351650b-9731-4e0b-a025-e65b6a1633ff-crio-socket\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:39.949597 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.949514 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4rt\" (UniqueName: \"kubernetes.io/projected/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-api-access-5p4rt\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:39.949597 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.949579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3351650b-9731-4e0b-a025-e65b6a1633ff-data-volume\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:39.949838 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:39.949600 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 11:37:39.949838 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:39.949603 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 11:37:39.949838 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:39.949669 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls podName:3351650b-9731-4e0b-a025-e65b6a1633ff nodeName:}" failed. No retries permitted until 2026-09-15 11:37:40.44965338 +0000 UTC m=+47.679579894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-599wc" (UID: "3351650b-9731-4e0b-a025-e65b6a1633ff") : secret "insights-runtime-extractor-tls" not found Sep 15 11:37:39.949838 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:39.949703 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-rbac-proxy-cm podName:3351650b-9731-4e0b-a025-e65b6a1633ff nodeName:}" failed. No retries permitted until 2026-09-15 11:37:40.449684075 +0000 UTC m=+47.679610602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-599wc" (UID: "3351650b-9731-4e0b-a025-e65b6a1633ff") : configmap "kube-rbac-proxy" not found Sep 15 11:37:39.949976 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.949945 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3351650b-9731-4e0b-a025-e65b6a1633ff-data-volume\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:39.960115 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:39.960094 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p4rt\" (UniqueName: \"kubernetes.io/projected/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-api-access-5p4rt\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:40.453793 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:40.453749 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:40.453793 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:40.453797 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:40.454046 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:40.453913 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 11:37:40.454046 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:40.453937 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 11:37:40.454046 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:40.454003 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-rbac-proxy-cm podName:3351650b-9731-4e0b-a025-e65b6a1633ff nodeName:}" failed. No retries permitted until 2026-09-15 11:37:41.453980684 +0000 UTC m=+48.683907209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-599wc" (UID: "3351650b-9731-4e0b-a025-e65b6a1633ff") : configmap "kube-rbac-proxy" not found Sep 15 11:37:40.454046 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:40.454042 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls podName:3351650b-9731-4e0b-a025-e65b6a1633ff nodeName:}" failed. No retries permitted until 2026-09-15 11:37:41.454012094 +0000 UTC m=+48.683938615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-599wc" (UID: "3351650b-9731-4e0b-a025-e65b6a1633ff") : secret "insights-runtime-extractor-tls" not found Sep 15 11:37:40.536836 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:40.536810 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 11:37:40.537301 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:40.537243 2573 scope.go:117] "RemoveContainer" containerID="8931f8ab1c6333c73b9acff42a12088019dd324a2316f1438bfc4a343d92380c" Sep 15 11:37:40.537446 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:40.537423 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-nzs6t_openshift-console-operator(590c2c95-948f-4758-8c73-c327a005adf1)\"" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" podUID="590c2c95-948f-4758-8c73-c327a005adf1" Sep 15 11:37:41.463425 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:41.463392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:41.463425 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:41.463430 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:41.463624 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:41.463547 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 11:37:41.463624 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:41.463572 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 11:37:41.463624 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:41.463622 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls podName:3351650b-9731-4e0b-a025-e65b6a1633ff nodeName:}" failed. No retries permitted until 2026-09-15 11:37:43.463603466 +0000 UTC m=+50.693529998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-599wc" (UID: "3351650b-9731-4e0b-a025-e65b6a1633ff") : secret "insights-runtime-extractor-tls" not found Sep 15 11:37:41.463723 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:41.463638 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-rbac-proxy-cm podName:3351650b-9731-4e0b-a025-e65b6a1633ff nodeName:}" failed. No retries permitted until 2026-09-15 11:37:43.463630902 +0000 UTC m=+50.693557416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-599wc" (UID: "3351650b-9731-4e0b-a025-e65b6a1633ff") : configmap "kube-rbac-proxy" not found Sep 15 11:37:41.572123 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:41.572097 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z8grp_b4cec296-04c2-48ae-8024-09e47e456b59/dns-node-resolver/0.log" Sep 15 11:37:42.172440 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:42.172413 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2sll8_14fd92d1-ab33-45f3-b40f-10048dda0c33/node-ca/0.log" Sep 15 11:37:42.972778 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:42.972753 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hp6hw_a645fa8f-dad2-486b-b3af-7ee707deaa93/serve-healthcheck-canary/0.log" Sep 15 11:37:43.177532 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:43.177494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q58lq\" (UID: \"dee2b36b-0dc3-4848-a413-ffca15516920\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:37:43.177717 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:43.177669 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 11:37:43.177781 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:43.177754 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert podName:dee2b36b-0dc3-4848-a413-ffca15516920 nodeName:}" failed. No retries permitted until 2026-09-15 11:37:51.177732482 +0000 UTC m=+58.407659015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-q58lq" (UID: "dee2b36b-0dc3-4848-a413-ffca15516920") : secret "networking-console-plugin-cert" not found Sep 15 11:37:43.479974 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:43.479932 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:43.480197 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:43.479976 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:43.480197 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:43.480126 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 11:37:43.480292 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:43.480206 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls podName:3351650b-9731-4e0b-a025-e65b6a1633ff nodeName:}" failed. No retries permitted until 2026-09-15 11:37:47.480189627 +0000 UTC m=+54.710116141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-599wc" (UID: "3351650b-9731-4e0b-a025-e65b6a1633ff") : secret "insights-runtime-extractor-tls" not found Sep 15 11:37:43.480506 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:43.480486 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3351650b-9731-4e0b-a025-e65b6a1633ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:44.084667 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:44.084630 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:37:44.085060 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:44.084700 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:37:44.085060 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:44.084775 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 11:37:44.085060 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:44.084822 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 11:37:44.085060 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:44.084833 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf4d4794-zt4c6: secret "image-registry-tls" not found Sep 15 11:37:44.085060 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:44.084838 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls podName:bbdaac6c-adf2-43e0-ae77-954292189938 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:00.084823597 +0000 UTC m=+67.314750115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-n6tc5" (UID: "bbdaac6c-adf2-43e0-ae77-954292189938") : secret "cluster-monitoring-operator-tls" not found Sep 15 11:37:44.085060 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:44.084869 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls podName:ed2b367e-e74c-4a65-92a7-ea170e652aae nodeName:}" failed. No retries permitted until 2026-09-15 11:38:00.084857823 +0000 UTC m=+67.314784337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls") pod "image-registry-7cf4d4794-zt4c6" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae") : secret "image-registry-tls" not found Sep 15 11:37:44.186059 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:44.186007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:44.186203 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:44.186105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:37:44.186203 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:44.186145 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 11:37:44.186203 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:44.186156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9slkd\" (UID: \"a247eace-c065-4b91-b2e5-a2d0c88cae08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:37:44.186336 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:44.186206 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs podName:78d07596-2598-4c06-a63c-ecb48c2052a0 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:00.186192011 +0000 UTC m=+67.416118525 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs") pod "router-default-5ddcf9f498-ghttm" (UID: "78d07596-2598-4c06-a63c-ecb48c2052a0") : secret "router-metrics-certs-default" not found Sep 15 11:37:44.186336 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:44.186236 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 11:37:44.186336 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:44.186239 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle podName:78d07596-2598-4c06-a63c-ecb48c2052a0 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:00.186227146 +0000 UTC m=+67.416153659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle") pod "router-default-5ddcf9f498-ghttm" (UID: "78d07596-2598-4c06-a63c-ecb48c2052a0") : configmap references non-existent config key: service-ca.crt Sep 15 11:37:44.186336 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:44.186278 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls podName:a247eace-c065-4b91-b2e5-a2d0c88cae08 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:00.186269211 +0000 UTC m=+67.416195726 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9slkd" (UID: "a247eace-c065-4b91-b2e5-a2d0c88cae08") : secret "samples-operator-tls" not found Sep 15 11:37:44.286820 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:44.286784 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:37:44.286973 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:44.286921 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 11:37:44.286973 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:44.286969 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls podName:39b7b0de-0d7f-4438-bd33-6a189e691139 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:00.286958163 +0000 UTC m=+67.516884677 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls") pod "dns-default-94pfg" (UID: "39b7b0de-0d7f-4438-bd33-6a189e691139") : secret "dns-default-metrics-tls" not found Sep 15 11:37:47.515088 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:47.515037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:47.515678 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:47.515191 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 11:37:47.515678 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:47.515282 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls podName:3351650b-9731-4e0b-a025-e65b6a1633ff nodeName:}" failed. No retries permitted until 2026-09-15 11:37:55.5152598 +0000 UTC m=+62.745186333 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-599wc" (UID: "3351650b-9731-4e0b-a025-e65b6a1633ff") : secret "insights-runtime-extractor-tls" not found Sep 15 11:37:48.663841 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:48.663798 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:48.664252 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:48.663863 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:37:48.664252 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:48.664215 2573 scope.go:117] "RemoveContainer" containerID="8931f8ab1c6333c73b9acff42a12088019dd324a2316f1438bfc4a343d92380c" Sep 15 11:37:48.664378 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:48.664358 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-nzs6t_openshift-console-operator(590c2c95-948f-4758-8c73-c327a005adf1)\"" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" podUID="590c2c95-948f-4758-8c73-c327a005adf1" Sep 15 11:37:51.250132 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:51.250095 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q58lq\" (UID: \"dee2b36b-0dc3-4848-a413-ffca15516920\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:37:51.250475 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:51.250203 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 11:37:51.250475 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:37:51.250250 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert podName:dee2b36b-0dc3-4848-a413-ffca15516920 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:07.250237835 +0000 UTC m=+74.480164348 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-q58lq" (UID: "dee2b36b-0dc3-4848-a413-ffca15516920") : secret "networking-console-plugin-cert" not found Sep 15 11:37:53.448986 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:53.448956 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kj4f2" Sep 15 11:37:55.588229 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:55.588195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:55.590459 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:55.590440 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3351650b-9731-4e0b-a025-e65b6a1633ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-599wc\" (UID: \"3351650b-9731-4e0b-a025-e65b6a1633ff\") " pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:55.734233 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:55.734199 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6nc5j\"" Sep 15 11:37:55.742550 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:55.742529 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-599wc" Sep 15 11:37:56.013168 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:56.012978 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-599wc"] Sep 15 11:37:56.015516 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:37:56.015491 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3351650b_9731_4e0b_a025_e65b6a1633ff.slice/crio-b75c849520a8834a293e1b7b12695b2cfdda76497ef5b310cd259017c394d10a WatchSource:0}: Error finding container b75c849520a8834a293e1b7b12695b2cfdda76497ef5b310cd259017c394d10a: Status 404 returned error can't find the container with id b75c849520a8834a293e1b7b12695b2cfdda76497ef5b310cd259017c394d10a Sep 15 11:37:56.588523 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:56.588487 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-599wc" event={"ID":"3351650b-9731-4e0b-a025-e65b6a1633ff","Type":"ContainerStarted","Data":"2bb8c73befe0a9586c5304504e0195a2f66d25c6b37e744d95a2b64e9191c16a"} Sep 15 11:37:56.588889 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:56.588545 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-599wc" event={"ID":"3351650b-9731-4e0b-a025-e65b6a1633ff","Type":"ContainerStarted","Data":"b75c849520a8834a293e1b7b12695b2cfdda76497ef5b310cd259017c394d10a"} Sep 15 11:37:57.595280 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:57.595237 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-599wc" event={"ID":"3351650b-9731-4e0b-a025-e65b6a1633ff","Type":"ContainerStarted","Data":"7a57259db4371625459b1874702de2c73d3b087212572bcc571eadd28097f3ac"} Sep 15 11:37:58.600459 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:58.600421 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-599wc" event={"ID":"3351650b-9731-4e0b-a025-e65b6a1633ff","Type":"ContainerStarted","Data":"ecc242cba5c8b4d3431579223339ada2f72d018b9b593855982b4ffcb59e6caf"} Sep 15 11:37:58.620543 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:58.620490 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-599wc" podStartSLOduration=17.537039318 podStartE2EDuration="19.620476639s" totalImagesPullingTime="2.083437321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:37:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:58.620153697 +0000 UTC m=+65.850080233" watchObservedRunningTime="2026-09-15 11:37:58.620476639 +0000 UTC m=+65.850403174" Sep 15 11:37:59.022446 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.022406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs\") pod \"network-metrics-daemon-ldmmr\" (UID: \"91f95d2b-d44c-4f8a-83bf-649275a608e7\") " pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:59.022626 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.022474 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:59.025154 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.025133 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 11:37:59.025256 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.025191 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 11:37:59.035236 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.035212 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecdd2ee0-6fba-4398-ad1b-87c1a4dca820-original-pull-secret\") pod \"global-pull-secret-syncer-msl5t\" (UID: \"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820\") " pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:59.035316 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.035213 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f95d2b-d44c-4f8a-83bf-649275a608e7-metrics-certs\") pod \"network-metrics-daemon-ldmmr\" (UID: \"91f95d2b-d44c-4f8a-83bf-649275a608e7\") " pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:59.123882 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.123840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxf9\" (UniqueName: \"kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9\") pod \"network-check-target-zdv99\" (UID: \"ffa5fd6d-96fc-4cd4-ad93-269a473cd51b\") " pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:59.126313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.126292 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jxf9\" (UniqueName: \"kubernetes.io/projected/ffa5fd6d-96fc-4cd4-ad93-269a473cd51b-kube-api-access-6jxf9\") pod \"network-check-target-zdv99\" (UID: \"ffa5fd6d-96fc-4cd4-ad93-269a473cd51b\") " pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:59.171645 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.171620 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-72bfh\"" Sep 15 11:37:59.179491 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.179472 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:59.179843 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.179829 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w6k5p\"" Sep 15 11:37:59.187424 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.187404 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmmr" Sep 15 11:37:59.239863 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.239837 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-msl5t" Sep 15 11:37:59.452219 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.452189 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zdv99"] Sep 15 11:37:59.455311 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:37:59.455284 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa5fd6d_96fc_4cd4_ad93_269a473cd51b.slice/crio-1198386924d6f4999201ea29bb53f8c0ed6c09befff75d5efd1a6a1750a16254 WatchSource:0}: Error finding container 1198386924d6f4999201ea29bb53f8c0ed6c09befff75d5efd1a6a1750a16254: Status 404 returned error can't find the container with id 1198386924d6f4999201ea29bb53f8c0ed6c09befff75d5efd1a6a1750a16254 Sep 15 11:37:59.476116 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.476094 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldmmr"] Sep 15 11:37:59.492050 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:37:59.492003 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f95d2b_d44c_4f8a_83bf_649275a608e7.slice/crio-7de1645c0220eee2db4788947db425ad6a86e5398136a862de149fbaaca00967 WatchSource:0}: Error finding container 7de1645c0220eee2db4788947db425ad6a86e5398136a862de149fbaaca00967: Status 404 returned error can't find the container with id 7de1645c0220eee2db4788947db425ad6a86e5398136a862de149fbaaca00967 Sep 15 11:37:59.516968 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.516524 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-msl5t"] Sep 15 11:37:59.521244 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:37:59.521220 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdd2ee0_6fba_4398_ad1b_87c1a4dca820.slice/crio-fba2abe9f82cf99e1da097f21e364eec68f3902e1c97e5261956a2e986504bfc WatchSource:0}: Error finding container fba2abe9f82cf99e1da097f21e364eec68f3902e1c97e5261956a2e986504bfc: Status 404 returned error can't find the container with id fba2abe9f82cf99e1da097f21e364eec68f3902e1c97e5261956a2e986504bfc Sep 15 11:37:59.604559 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.604470 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldmmr" event={"ID":"91f95d2b-d44c-4f8a-83bf-649275a608e7","Type":"ContainerStarted","Data":"7de1645c0220eee2db4788947db425ad6a86e5398136a862de149fbaaca00967"} Sep 15 11:37:59.605971 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.605950 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zdv99" event={"ID":"ffa5fd6d-96fc-4cd4-ad93-269a473cd51b","Type":"ContainerStarted","Data":"7e44b87bf57079eec6c8ebbe1cb25466e140040384e4ed52e9a090ef61565583"} Sep 15 11:37:59.606084 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.605990 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zdv99" event={"ID":"ffa5fd6d-96fc-4cd4-ad93-269a473cd51b","Type":"ContainerStarted","Data":"1198386924d6f4999201ea29bb53f8c0ed6c09befff75d5efd1a6a1750a16254"} Sep 15 11:37:59.606084 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.606062 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:37:59.607133 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.607114 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-msl5t" event={"ID":"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820","Type":"ContainerStarted","Data":"fba2abe9f82cf99e1da097f21e364eec68f3902e1c97e5261956a2e986504bfc"} Sep 15 11:37:59.623183 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:37:59.623146 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zdv99" podStartSLOduration=66.623133083 podStartE2EDuration="1m6.623133083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:37:59.62281915 +0000 UTC m=+66.852745685" watchObservedRunningTime="2026-09-15 11:37:59.623133083 +0000 UTC m=+66.853059619" Sep 15 11:38:00.133424 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.133383 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:38:00.133608 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.133521 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:38:00.136069 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.136042 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls\") pod \"image-registry-7cf4d4794-zt4c6\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:38:00.136177 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.136097 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdaac6c-adf2-43e0-ae77-954292189938-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-n6tc5\" (UID: \"bbdaac6c-adf2-43e0-ae77-954292189938\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:38:00.234966 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.234928 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:38:00.235146 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.235049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:38:00.235146 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.235105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9slkd\" (UID: \"a247eace-c065-4b91-b2e5-a2d0c88cae08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:38:00.235793 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.235765 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78d07596-2598-4c06-a63c-ecb48c2052a0-service-ca-bundle\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:38:00.237704 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.237672 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a247eace-c065-4b91-b2e5-a2d0c88cae08-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9slkd\" (UID: \"a247eace-c065-4b91-b2e5-a2d0c88cae08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:38:00.237926 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.237906 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d07596-2598-4c06-a63c-ecb48c2052a0-metrics-certs\") pod \"router-default-5ddcf9f498-ghttm\" (UID: \"78d07596-2598-4c06-a63c-ecb48c2052a0\") " pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:38:00.335519 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.335484 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:38:00.338274 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.338248 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b7b0de-0d7f-4438-bd33-6a189e691139-metrics-tls\") pod \"dns-default-94pfg\" (UID: \"39b7b0de-0d7f-4438-bd33-6a189e691139\") " pod="openshift-dns/dns-default-94pfg" Sep 15 11:38:00.343174 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.343150 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m5mwt\"" Sep 15 11:38:00.350497 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.350471 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" Sep 15 11:38:00.362735 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.362713 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ctgh2\"" Sep 15 11:38:00.371092 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.371072 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:38:00.395334 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.395233 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pvn7w\"" Sep 15 11:38:00.403458 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.403364 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:38:00.454037 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.449943 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zcsmb\"" Sep 15 11:38:00.458663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.458288 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" Sep 15 11:38:00.531646 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.531281 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qkkmw\"" Sep 15 11:38:00.538718 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.538680 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94pfg" Sep 15 11:38:00.633966 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.633934 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5"] Sep 15 11:38:00.666681 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.666642 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cf4d4794-zt4c6"] Sep 15 11:38:00.712801 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.712772 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5ddcf9f498-ghttm"] Sep 15 11:38:00.752507 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:00.752479 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd"] Sep 15 11:38:00.800560 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:38:00.800527 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2b367e_e74c_4a65_92a7_ea170e652aae.slice/crio-55a17a8d6482da10ac7cccba8f0449179a684771e05b731ba438b1be1be90a23 WatchSource:0}: Error finding container 55a17a8d6482da10ac7cccba8f0449179a684771e05b731ba438b1be1be90a23: Status 404 returned error can't find the container with id 55a17a8d6482da10ac7cccba8f0449179a684771e05b731ba438b1be1be90a23 Sep 15 11:38:00.802155 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:38:00.802114 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbdaac6c_adf2_43e0_ae77_954292189938.slice/crio-af48047fa9073d895b84f03f984bcd7e2082a609f4377710004d5e78e6b1df1a WatchSource:0}: Error finding container af48047fa9073d895b84f03f984bcd7e2082a609f4377710004d5e78e6b1df1a: Status 404 returned error can't find the container with id af48047fa9073d895b84f03f984bcd7e2082a609f4377710004d5e78e6b1df1a Sep 15 11:38:01.080673 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.080636 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94pfg"] Sep 15 11:38:01.098614 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:38:01.098576 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b7b0de_0d7f_4438_bd33_6a189e691139.slice/crio-712bb3fabd763ff045373cf187700268fe4a541893dec6787ab1c4991fa80a10 WatchSource:0}: Error finding container 712bb3fabd763ff045373cf187700268fe4a541893dec6787ab1c4991fa80a10: Status 404 returned error can't find the container with id 712bb3fabd763ff045373cf187700268fe4a541893dec6787ab1c4991fa80a10 Sep 15 11:38:01.232040 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.231546 2573 scope.go:117] "RemoveContainer" containerID="8931f8ab1c6333c73b9acff42a12088019dd324a2316f1438bfc4a343d92380c" Sep 15 11:38:01.623038 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.622193 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldmmr" event={"ID":"91f95d2b-d44c-4f8a-83bf-649275a608e7","Type":"ContainerStarted","Data":"fc101dce88c61f5e47590a9e6ac2f16b0cc4ef9acdf64b7fd30b7956477f00ab"} Sep 15 11:38:01.623038 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.622261 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldmmr" event={"ID":"91f95d2b-d44c-4f8a-83bf-649275a608e7","Type":"ContainerStarted","Data":"237b2a97030016f0620170f6288515f4faedcf09960a540986ac784dee51ce7e"} Sep 15 11:38:01.624517 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.624451 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94pfg" event={"ID":"39b7b0de-0d7f-4438-bd33-6a189e691139","Type":"ContainerStarted","Data":"712bb3fabd763ff045373cf187700268fe4a541893dec6787ab1c4991fa80a10"} Sep 15 11:38:01.627179 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.627149 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" event={"ID":"a247eace-c065-4b91-b2e5-a2d0c88cae08","Type":"ContainerStarted","Data":"2a00fbab6d44c9c8eab6620a6adb812f8e9d353ace80f77b07c177c6fbdb5c2c"} Sep 15 11:38:01.629749 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.629253 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5ddcf9f498-ghttm" event={"ID":"78d07596-2598-4c06-a63c-ecb48c2052a0","Type":"ContainerStarted","Data":"678574fad6d62603c6b2085e44b85212f8c102336a70e32cc795a23e086eaf0d"} Sep 15 11:38:01.629749 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.629294 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5ddcf9f498-ghttm" event={"ID":"78d07596-2598-4c06-a63c-ecb48c2052a0","Type":"ContainerStarted","Data":"122f27c17ce22310465b3f8c2f4d7040e537c38197c106a1674a65e483631d5d"} Sep 15 11:38:01.630956 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.630925 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" event={"ID":"bbdaac6c-adf2-43e0-ae77-954292189938","Type":"ContainerStarted","Data":"af48047fa9073d895b84f03f984bcd7e2082a609f4377710004d5e78e6b1df1a"} Sep 15 11:38:01.634956 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.634148 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 11:38:01.634956 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.634225 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" event={"ID":"590c2c95-948f-4758-8c73-c327a005adf1","Type":"ContainerStarted","Data":"c161e3d26652b6117c79edde70c68fabe8ac3ceb177dbedacfcdbff59b02db59"} Sep 15 11:38:01.634956 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.634915 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:38:01.637954 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.637924 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" event={"ID":"ed2b367e-e74c-4a65-92a7-ea170e652aae","Type":"ContainerStarted","Data":"96523da2005e3f9650dedcba21c08d4a66fc25703ee8885baa5ae75facd64445"} Sep 15 11:38:01.638180 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.637975 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" event={"ID":"ed2b367e-e74c-4a65-92a7-ea170e652aae","Type":"ContainerStarted","Data":"55a17a8d6482da10ac7cccba8f0449179a684771e05b731ba438b1be1be90a23"} Sep 15 11:38:01.638458 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.638435 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:38:01.640976 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.640953 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" Sep 15 11:38:01.644992 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.644946 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ldmmr" podStartSLOduration=67.028508615 podStartE2EDuration="1m8.644901152s" totalImagesPullingTime="1.616392537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:01.643472293 +0000 UTC m=+68.873398826" watchObservedRunningTime="2026-09-15 11:38:01.644901152 +0000 UTC m=+68.874827716" Sep 15 11:38:01.665544 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.665497 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" podStartSLOduration=68.665479068 podStartE2EDuration="1m8.665479068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:01.66399014 +0000 UTC m=+68.893916677" watchObservedRunningTime="2026-09-15 11:38:01.665479068 +0000 UTC m=+68.895405605" Sep 15 11:38:01.717312 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.716422 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5ddcf9f498-ghttm" podStartSLOduration=56.716405479 podStartE2EDuration="56.716405479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:01.685945137 +0000 UTC m=+68.915871674" watchObservedRunningTime="2026-09-15 11:38:01.716405479 +0000 UTC m=+68.946332015" Sep 15 11:38:01.717312 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:01.716609 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-nzs6t" podStartSLOduration=48.095724994 podStartE2EDuration="55.71660053s" totalImagesPullingTime="7.620875536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:01.714975691 +0000 UTC m=+68.944902229" watchObservedRunningTime="2026-09-15 11:38:01.71660053 +0000 UTC m=+68.946527066" Sep 15 11:38:02.404470 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:02.404427 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:38:02.407302 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:02.407275 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:38:02.643110 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:02.643003 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:38:02.644488 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:02.644465 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5ddcf9f498-ghttm" Sep 15 11:38:03.728119 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.728079 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8"] Sep 15 11:38:03.731527 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.731500 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.735123 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.735101 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 15 11:38:03.735624 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.735600 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 15 11:38:03.735756 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.735607 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 15 11:38:03.735756 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.735644 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 15 11:38:03.735856 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.735798 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 15 11:38:03.735897 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.735861 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 15 11:38:03.743351 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.743327 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 15 11:38:03.752839 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.752798 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8"] Sep 15 11:38:03.874806 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.874769 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f53a34-566c-4d1b-bfa6-e5abf082805b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.874972 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.874819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/84f53a34-566c-4d1b-bfa6-e5abf082805b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.874972 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.874859 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/84f53a34-566c-4d1b-bfa6-e5abf082805b-hub\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.874972 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.874948 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84f53a34-566c-4d1b-bfa6-e5abf082805b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.875149 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.875007 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/84f53a34-566c-4d1b-bfa6-e5abf082805b-ca\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.875149 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.875064 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hb7h\" (UniqueName: \"kubernetes.io/projected/84f53a34-566c-4d1b-bfa6-e5abf082805b-kube-api-access-7hb7h\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.975760 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.975724 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f53a34-566c-4d1b-bfa6-e5abf082805b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.975760 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.975762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/84f53a34-566c-4d1b-bfa6-e5abf082805b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.976014 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.975795 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/84f53a34-566c-4d1b-bfa6-e5abf082805b-hub\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.976014 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.975835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84f53a34-566c-4d1b-bfa6-e5abf082805b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.976014 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.975879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/84f53a34-566c-4d1b-bfa6-e5abf082805b-ca\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.976014 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.975921 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hb7h\" (UniqueName: \"kubernetes.io/projected/84f53a34-566c-4d1b-bfa6-e5abf082805b-kube-api-access-7hb7h\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.976537 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.976503 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f53a34-566c-4d1b-bfa6-e5abf082805b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.979190 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.979119 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/84f53a34-566c-4d1b-bfa6-e5abf082805b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.979295 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.979185 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84f53a34-566c-4d1b-bfa6-e5abf082805b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.979560 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.979539 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/84f53a34-566c-4d1b-bfa6-e5abf082805b-ca\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.979741 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.979713 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/84f53a34-566c-4d1b-bfa6-e5abf082805b-hub\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:03.988280 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:03.988258 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hb7h\" (UniqueName: \"kubernetes.io/projected/84f53a34-566c-4d1b-bfa6-e5abf082805b-kube-api-access-7hb7h\") pod \"cluster-proxy-proxy-agent-77cb849494-b7dt8\" (UID: \"84f53a34-566c-4d1b-bfa6-e5abf082805b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:04.050452 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:04.050419 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" Sep 15 11:38:05.490054 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:05.490005 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8"] Sep 15 11:38:05.495869 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:38:05.495825 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84f53a34_566c_4d1b_bfa6_e5abf082805b.slice/crio-81522cb9d5b03a9394130a5513262ee62b2ef34b2790c256a623296a5994e772 WatchSource:0}: Error finding container 81522cb9d5b03a9394130a5513262ee62b2ef34b2790c256a623296a5994e772: Status 404 returned error can't find the container with id 81522cb9d5b03a9394130a5513262ee62b2ef34b2790c256a623296a5994e772 Sep 15 11:38:05.655844 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:05.655806 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94pfg" event={"ID":"39b7b0de-0d7f-4438-bd33-6a189e691139","Type":"ContainerStarted","Data":"6e26397b5938e32e60e4ffb175087254146fc46de6e82b26af7e26b5edcfe38e"} Sep 15 11:38:05.656017 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:05.655912 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94pfg" event={"ID":"39b7b0de-0d7f-4438-bd33-6a189e691139","Type":"ContainerStarted","Data":"ca1d9c04782d42dbbdaab77b8274d6ddcf4347b48285af68abf5d501244e00d5"} Sep 15 11:38:05.656017 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:05.655944 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-94pfg" Sep 15 11:38:05.657832 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:05.657807 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" event={"ID":"a247eace-c065-4b91-b2e5-a2d0c88cae08","Type":"ContainerStarted","Data":"da5ce65c37333a5d452f48af623cf6031210a2f57b30fe178a0c46e69373102c"} Sep 15 11:38:05.657939 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:05.657851 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" event={"ID":"a247eace-c065-4b91-b2e5-a2d0c88cae08","Type":"ContainerStarted","Data":"2e17622c1afe2db1818bc02316695461fa4a5206466fa9f74305285f633f036e"} Sep 15 11:38:05.659377 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:05.659350 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-msl5t" event={"ID":"ecdd2ee0-6fba-4398-ad1b-87c1a4dca820","Type":"ContainerStarted","Data":"8a1712f6d1ea1dfa81ca13eeedea1ff27a991265c6ea5b2a69105942ec29cf8d"} Sep 15 11:38:05.660949 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:05.660920 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" event={"ID":"bbdaac6c-adf2-43e0-ae77-954292189938","Type":"ContainerStarted","Data":"7a12fc9f0a3ae2a89087206e089fb2f4317d4bc0157248820abbe157e74625e8"} Sep 15 11:38:05.662124 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:05.662105 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" event={"ID":"84f53a34-566c-4d1b-bfa6-e5abf082805b","Type":"ContainerStarted","Data":"81522cb9d5b03a9394130a5513262ee62b2ef34b2790c256a623296a5994e772"} Sep 15 11:38:05.677086 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:05.677013 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-94pfg" podStartSLOduration=33.578154802 podStartE2EDuration="37.676998985s" totalImagesPullingTime="4.098844183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:37:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:05.6750055 +0000 UTC m=+72.904932037" watchObservedRunningTime="2026-09-15 11:38:05.676998985 +0000 UTC m=+72.906925520" Sep 15 11:38:05.694342 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:05.694301 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-n6tc5" podStartSLOduration=56.295895721 podStartE2EDuration="1m0.694288038s" totalImagesPullingTime="4.398392317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:05.693922417 +0000 UTC m=+72.923848954" watchObservedRunningTime="2026-09-15 11:38:05.694288038 +0000 UTC m=+72.924214574" Sep 15 11:38:05.727059 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:05.726951 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-msl5t" podStartSLOduration=67.035830216 podStartE2EDuration="1m12.726934836s" totalImagesPullingTime="5.69110462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:05.709018321 +0000 UTC m=+72.938944857" watchObservedRunningTime="2026-09-15 11:38:05.726934836 +0000 UTC m=+72.956861362" Sep 15 11:38:05.727220 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:05.727186 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9slkd" podStartSLOduration=55.393368517 podStartE2EDuration="59.727173578s" totalImagesPullingTime="4.333805061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:05.725978339 +0000 UTC m=+72.955904877" watchObservedRunningTime="2026-09-15 11:38:05.727173578 +0000 UTC m=+72.957100116" Sep 15 11:38:07.307928 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:07.307888 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q58lq\" (UID: \"dee2b36b-0dc3-4848-a413-ffca15516920\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:38:07.310691 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:07.310656 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dee2b36b-0dc3-4848-a413-ffca15516920-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-q58lq\" (UID: \"dee2b36b-0dc3-4848-a413-ffca15516920\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:38:07.520292 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:07.520261 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7vb4l\"" Sep 15 11:38:07.528195 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:07.528165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" Sep 15 11:38:08.007769 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:08.007738 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq"] Sep 15 11:38:09.075351 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:38:09.075321 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee2b36b_0dc3_4848_a413_ffca15516920.slice/crio-4a819f86a9bf09fad77a641e32d73cb117d6890b60a30bf0bdb9398216f4fc82 WatchSource:0}: Error finding container 4a819f86a9bf09fad77a641e32d73cb117d6890b60a30bf0bdb9398216f4fc82: Status 404 returned error can't find the container with id 4a819f86a9bf09fad77a641e32d73cb117d6890b60a30bf0bdb9398216f4fc82 Sep 15 11:38:09.680763 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:09.680735 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" event={"ID":"84f53a34-566c-4d1b-bfa6-e5abf082805b","Type":"ContainerStarted","Data":"607e9139853089433ac6682161bd8e696c342c84a3cfe0868f8205c0275dacc9"} Sep 15 11:38:09.682036 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:09.682000 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" event={"ID":"dee2b36b-0dc3-4848-a413-ffca15516920","Type":"ContainerStarted","Data":"4a819f86a9bf09fad77a641e32d73cb117d6890b60a30bf0bdb9398216f4fc82"} Sep 15 11:38:10.688680 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:10.688653 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" event={"ID":"84f53a34-566c-4d1b-bfa6-e5abf082805b","Type":"ContainerStarted","Data":"c33c28db476f3faeba69836ef3b7e8c0fdbb1d43249d186eb0961f2ebc56acfa"} Sep 15 11:38:10.689060 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:10.688694 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" event={"ID":"84f53a34-566c-4d1b-bfa6-e5abf082805b","Type":"ContainerStarted","Data":"02d77ca1a06821bd504e56dbd4af4101f8668f5e7f1ab4e01fcef2e3d409b5d2"} Sep 15 11:38:10.709428 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:10.709392 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cb849494-b7dt8" podStartSLOduration=3.70304119 podStartE2EDuration="7.709377358s" totalImagesPullingTime="4.006336168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:10.708293763 +0000 UTC m=+77.938220312" watchObservedRunningTime="2026-09-15 11:38:10.709377358 +0000 UTC m=+77.939303914" Sep 15 11:38:11.694253 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:11.694214 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" event={"ID":"dee2b36b-0dc3-4848-a413-ffca15516920","Type":"ContainerStarted","Data":"2718e8ec65c1d876cd07b4d2322389edd0a4204cde6123a8295592e001698fb2"} Sep 15 11:38:11.710972 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:11.710927 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-q58lq" podStartSLOduration=35.147463998 podStartE2EDuration="36.710908822s" totalImagesPullingTime="1.563444824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:11.709477468 +0000 UTC m=+78.939404003" watchObservedRunningTime="2026-09-15 11:38:11.710908822 +0000 UTC m=+78.940835359" Sep 15 11:38:15.669129 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:15.669098 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-94pfg" Sep 15 11:38:19.508541 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.508508 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pngt7"] Sep 15 11:38:19.511278 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.511258 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.515672 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.515650 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 15 11:38:19.515924 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.515903 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 15 11:38:19.515984 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.515932 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-842fw\"" Sep 15 11:38:19.516197 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.516183 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 15 11:38:19.521697 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.521681 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 15 11:38:19.604342 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.604301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6d1bd46-92b5-4844-9d77-932851964761-sys\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.604342 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.604348 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-textfile\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.604550 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.604420 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.604550 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.604454 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-accelerators-collector-config\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.604550 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.604472 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtp55\" (UniqueName: \"kubernetes.io/projected/e6d1bd46-92b5-4844-9d77-932851964761-kube-api-access-xtp55\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.604550 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.604509 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6d1bd46-92b5-4844-9d77-932851964761-metrics-client-ca\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.604703 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.604555 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6d1bd46-92b5-4844-9d77-932851964761-root\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.604703 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.604573 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-wtmp\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.604703 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.604600 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-tls\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.705824 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.705780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-tls\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.705997 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.705833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6d1bd46-92b5-4844-9d77-932851964761-sys\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.705997 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.705896 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6d1bd46-92b5-4844-9d77-932851964761-sys\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.705997 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:38:19.705935 2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 15 11:38:19.705997 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.705949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-textfile\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.705997 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:38:19.705996 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-tls podName:e6d1bd46-92b5-4844-9d77-932851964761 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:20.205980622 +0000 UTC m=+87.435907149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-tls") pod "node-exporter-pngt7" (UID: "e6d1bd46-92b5-4844-9d77-932851964761") : secret "node-exporter-tls" not found Sep 15 11:38:19.706320 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.706040 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.706320 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.706076 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-accelerators-collector-config\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.706320 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.706106 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtp55\" (UniqueName: \"kubernetes.io/projected/e6d1bd46-92b5-4844-9d77-932851964761-kube-api-access-xtp55\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.706320 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.706145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6d1bd46-92b5-4844-9d77-932851964761-metrics-client-ca\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.706320 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.706169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6d1bd46-92b5-4844-9d77-932851964761-root\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.706320 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.706199 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-wtmp\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.706320 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.706258 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6d1bd46-92b5-4844-9d77-932851964761-root\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.706320 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.706294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-textfile\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.706614 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.706361 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-wtmp\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.706693 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.706670 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-accelerators-collector-config\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.706732 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.706697 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6d1bd46-92b5-4844-9d77-932851964761-metrics-client-ca\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.708320 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.708297 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:19.715614 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:19.715579 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtp55\" (UniqueName: \"kubernetes.io/projected/e6d1bd46-92b5-4844-9d77-932851964761-kube-api-access-xtp55\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:20.210661 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:20.210623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-tls\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:20.212983 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:20.212959 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6d1bd46-92b5-4844-9d77-932851964761-node-exporter-tls\") pod \"node-exporter-pngt7\" (UID: \"e6d1bd46-92b5-4844-9d77-932851964761\") " pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:20.420239 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:20.420206 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pngt7" Sep 15 11:38:20.431234 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:38:20.431211 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d1bd46_92b5_4844_9d77_932851964761.slice/crio-e8e06566bf1a31db217c4f5f7d7c5a141500eb190ca3829281d66b389ce09b2f WatchSource:0}: Error finding container e8e06566bf1a31db217c4f5f7d7c5a141500eb190ca3829281d66b389ce09b2f: Status 404 returned error can't find the container with id e8e06566bf1a31db217c4f5f7d7c5a141500eb190ca3829281d66b389ce09b2f Sep 15 11:38:20.729502 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:20.729458 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pngt7" event={"ID":"e6d1bd46-92b5-4844-9d77-932851964761","Type":"ContainerStarted","Data":"e8e06566bf1a31db217c4f5f7d7c5a141500eb190ca3829281d66b389ce09b2f"} Sep 15 11:38:21.734405 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:21.734325 2573 generic.go:356] "Generic (PLEG): container finished" podID="e6d1bd46-92b5-4844-9d77-932851964761" containerID="35b4ec6962a52d9348d35c169ffe7101f9e1a2b61ad2b8d108f54436b05c81cd" exitCode=0 Sep 15 11:38:21.734748 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:21.734418 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pngt7" event={"ID":"e6d1bd46-92b5-4844-9d77-932851964761","Type":"ContainerDied","Data":"35b4ec6962a52d9348d35c169ffe7101f9e1a2b61ad2b8d108f54436b05c81cd"} Sep 15 11:38:22.740594 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:22.740558 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pngt7" event={"ID":"e6d1bd46-92b5-4844-9d77-932851964761","Type":"ContainerStarted","Data":"8cdf1723ee036c0cd7654ced9ae8a6ee5c3a44c8df679180497402426dc055bf"} Sep 15 11:38:22.741058 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:22.740607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pngt7" event={"ID":"e6d1bd46-92b5-4844-9d77-932851964761","Type":"ContainerStarted","Data":"9ad04f38e47bbc31029fa6c18b3fe807e831f67be892e92acb0e1c4b13870d79"} Sep 15 11:38:22.767717 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:22.767674 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pngt7" podStartSLOduration=2.662384233 podStartE2EDuration="3.767660108s" totalImagesPullingTime="1.042337727s" totalInitContainerRuntime="62.938148ms" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:22.766453805 +0000 UTC m=+89.996380341" watchObservedRunningTime="2026-09-15 11:38:22.767660108 +0000 UTC m=+89.997586644" Sep 15 11:38:23.651515 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:23.651488 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:38:27.233712 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:27.233681 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cf4d4794-zt4c6"] Sep 15 11:38:30.618144 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:30.618111 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zdv99" Sep 15 11:38:44.824111 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:44.824078 2573 generic.go:356] "Generic (PLEG): container finished" podID="49259c18-d5c4-4a14-b715-60dae1b8a2ef" containerID="f00e2c76b3b173ee673ac1742221132808270a30df3f756b51d2c0080db9825f" exitCode=0 Sep 15 11:38:44.824606 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:44.824155 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" event={"ID":"49259c18-d5c4-4a14-b715-60dae1b8a2ef","Type":"ContainerDied","Data":"f00e2c76b3b173ee673ac1742221132808270a30df3f756b51d2c0080db9825f"} Sep 15 11:38:44.824606 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:44.824537 2573 scope.go:117] "RemoveContainer" containerID="f00e2c76b3b173ee673ac1742221132808270a30df3f756b51d2c0080db9825f" Sep 15 11:38:45.829796 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:45.829758 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6f89j" event={"ID":"49259c18-d5c4-4a14-b715-60dae1b8a2ef","Type":"ContainerStarted","Data":"a4fe53518f8d932461ea8e4dbcf1d47937f6b08fc91e2a0ba4a4f533434091c2"} Sep 15 11:38:52.252893 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.252852 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" podUID="ed2b367e-e74c-4a65-92a7-ea170e652aae" containerName="registry" containerID="cri-o://96523da2005e3f9650dedcba21c08d4a66fc25703ee8885baa5ae75facd64445" gracePeriod=30 Sep 15 11:38:52.488853 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.488827 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:38:52.571214 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.571137 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-certificates\") pod \"ed2b367e-e74c-4a65-92a7-ea170e652aae\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " Sep 15 11:38:52.571214 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.571202 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-bound-sa-token\") pod \"ed2b367e-e74c-4a65-92a7-ea170e652aae\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " Sep 15 11:38:52.571214 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.571222 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed2b367e-e74c-4a65-92a7-ea170e652aae-trusted-ca\") pod \"ed2b367e-e74c-4a65-92a7-ea170e652aae\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " Sep 15 11:38:52.571466 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.571338 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls\") pod \"ed2b367e-e74c-4a65-92a7-ea170e652aae\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " Sep 15 11:38:52.571466 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.571377 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed2b367e-e74c-4a65-92a7-ea170e652aae-installation-pull-secrets\") pod \"ed2b367e-e74c-4a65-92a7-ea170e652aae\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " Sep 15 11:38:52.571466 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.571402 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbf9f\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-kube-api-access-qbf9f\") pod \"ed2b367e-e74c-4a65-92a7-ea170e652aae\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " Sep 15 11:38:52.571466 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.571426 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed2b367e-e74c-4a65-92a7-ea170e652aae-image-registry-private-configuration\") pod \"ed2b367e-e74c-4a65-92a7-ea170e652aae\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " Sep 15 11:38:52.571466 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.571462 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed2b367e-e74c-4a65-92a7-ea170e652aae-ca-trust-extracted\") pod \"ed2b367e-e74c-4a65-92a7-ea170e652aae\" (UID: \"ed2b367e-e74c-4a65-92a7-ea170e652aae\") " Sep 15 11:38:52.571710 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.571596 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ed2b367e-e74c-4a65-92a7-ea170e652aae" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:38:52.571710 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.571608 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2b367e-e74c-4a65-92a7-ea170e652aae-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ed2b367e-e74c-4a65-92a7-ea170e652aae" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:38:52.571811 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.571731 2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-certificates\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:38:52.571811 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.571751 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed2b367e-e74c-4a65-92a7-ea170e652aae-trusted-ca\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:38:52.573833 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.573802 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2b367e-e74c-4a65-92a7-ea170e652aae-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ed2b367e-e74c-4a65-92a7-ea170e652aae" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:38:52.574126 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.574096 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ed2b367e-e74c-4a65-92a7-ea170e652aae" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:38:52.574126 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.574105 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2b367e-e74c-4a65-92a7-ea170e652aae-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ed2b367e-e74c-4a65-92a7-ea170e652aae" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:38:52.574296 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.574138 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ed2b367e-e74c-4a65-92a7-ea170e652aae" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:38:52.574296 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.574200 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-kube-api-access-qbf9f" (OuterVolumeSpecName: "kube-api-access-qbf9f") pod "ed2b367e-e74c-4a65-92a7-ea170e652aae" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae"). InnerVolumeSpecName "kube-api-access-qbf9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:38:52.582632 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.582605 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2b367e-e74c-4a65-92a7-ea170e652aae-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ed2b367e-e74c-4a65-92a7-ea170e652aae" (UID: "ed2b367e-e74c-4a65-92a7-ea170e652aae"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:38:52.672808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.672779 2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-bound-sa-token\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:38:52.672808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.672803 2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-registry-tls\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:38:52.672808 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.672812 2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed2b367e-e74c-4a65-92a7-ea170e652aae-installation-pull-secrets\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:38:52.672978 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.672822 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbf9f\" (UniqueName: \"kubernetes.io/projected/ed2b367e-e74c-4a65-92a7-ea170e652aae-kube-api-access-qbf9f\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:38:52.672978 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.672833 2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed2b367e-e74c-4a65-92a7-ea170e652aae-image-registry-private-configuration\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:38:52.672978 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.672842 2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed2b367e-e74c-4a65-92a7-ea170e652aae-ca-trust-extracted\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:38:52.856517 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.856419 2573 generic.go:356] "Generic (PLEG): container finished" podID="ed2b367e-e74c-4a65-92a7-ea170e652aae" containerID="96523da2005e3f9650dedcba21c08d4a66fc25703ee8885baa5ae75facd64445" exitCode=0 Sep 15 11:38:52.856517 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.856479 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" Sep 15 11:38:52.856727 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.856514 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" event={"ID":"ed2b367e-e74c-4a65-92a7-ea170e652aae","Type":"ContainerDied","Data":"96523da2005e3f9650dedcba21c08d4a66fc25703ee8885baa5ae75facd64445"} Sep 15 11:38:52.856727 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.856572 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf4d4794-zt4c6" event={"ID":"ed2b367e-e74c-4a65-92a7-ea170e652aae","Type":"ContainerDied","Data":"55a17a8d6482da10ac7cccba8f0449179a684771e05b731ba438b1be1be90a23"} Sep 15 11:38:52.856727 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.856596 2573 scope.go:117] "RemoveContainer" containerID="96523da2005e3f9650dedcba21c08d4a66fc25703ee8885baa5ae75facd64445" Sep 15 11:38:52.865319 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.865304 2573 scope.go:117] "RemoveContainer" containerID="96523da2005e3f9650dedcba21c08d4a66fc25703ee8885baa5ae75facd64445" Sep 15 11:38:52.865571 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:38:52.865553 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96523da2005e3f9650dedcba21c08d4a66fc25703ee8885baa5ae75facd64445\": container with ID starting with 96523da2005e3f9650dedcba21c08d4a66fc25703ee8885baa5ae75facd64445 not found: ID does not exist" containerID="96523da2005e3f9650dedcba21c08d4a66fc25703ee8885baa5ae75facd64445" Sep 15 11:38:52.865611 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.865578 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96523da2005e3f9650dedcba21c08d4a66fc25703ee8885baa5ae75facd64445"} err="failed to get container status \"96523da2005e3f9650dedcba21c08d4a66fc25703ee8885baa5ae75facd64445\": rpc error: code = NotFound desc = could not find container \"96523da2005e3f9650dedcba21c08d4a66fc25703ee8885baa5ae75facd64445\": container with ID starting with 96523da2005e3f9650dedcba21c08d4a66fc25703ee8885baa5ae75facd64445 not found: ID does not exist" Sep 15 11:38:52.877902 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.877882 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cf4d4794-zt4c6"] Sep 15 11:38:52.881599 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:52.881578 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cf4d4794-zt4c6"] Sep 15 11:38:53.235077 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:53.235049 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed2b367e-e74c-4a65-92a7-ea170e652aae" path="/var/lib/kubelet/pods/ed2b367e-e74c-4a65-92a7-ea170e652aae/volumes" Sep 15 11:38:54.867803 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:54.867765 2573 generic.go:356] "Generic (PLEG): container finished" podID="22eeb9ed-5ded-442e-82ae-ac31f979e681" containerID="b8beeb5491b7b32939186e22ee736dfb22fcf4402a0f4e2e94f956d8d5b08cba" exitCode=0 Sep 15 11:38:54.868171 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:54.867837 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hd9fz" event={"ID":"22eeb9ed-5ded-442e-82ae-ac31f979e681","Type":"ContainerDied","Data":"b8beeb5491b7b32939186e22ee736dfb22fcf4402a0f4e2e94f956d8d5b08cba"} Sep 15 11:38:54.868222 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:54.868208 2573 scope.go:117] "RemoveContainer" containerID="b8beeb5491b7b32939186e22ee736dfb22fcf4402a0f4e2e94f956d8d5b08cba" Sep 15 11:38:55.872967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:38:55.872936 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hd9fz" event={"ID":"22eeb9ed-5ded-442e-82ae-ac31f979e681","Type":"ContainerStarted","Data":"beefee3014cd2a8c116ee39ec6df46644116c18364e1936e5b7503f841a15036"} Sep 15 11:39:08.921417 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:39:08.921380 2573 generic.go:356] "Generic (PLEG): container finished" podID="76f77d07-00dd-441c-82ab-d037c4a6da6c" containerID="f195f52bcdc8bf8b38f261435a153e60b8f44638e2be9ca3718ac1a574446d9f" exitCode=0 Sep 15 11:39:08.921750 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:39:08.921447 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" event={"ID":"76f77d07-00dd-441c-82ab-d037c4a6da6c","Type":"ContainerDied","Data":"f195f52bcdc8bf8b38f261435a153e60b8f44638e2be9ca3718ac1a574446d9f"} Sep 15 11:39:08.921794 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:39:08.921759 2573 scope.go:117] "RemoveContainer" containerID="f195f52bcdc8bf8b38f261435a153e60b8f44638e2be9ca3718ac1a574446d9f" Sep 15 11:39:09.927128 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:39:09.927097 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4cfrz" event={"ID":"76f77d07-00dd-441c-82ab-d037c4a6da6c","Type":"ContainerStarted","Data":"7f1484b0a4026746f4e3f5a24a04a3f6e6c251211b7771415fb98ddc57792268"} Sep 15 11:41:47.298452 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.298414 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk"] Sep 15 11:41:47.299012 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.298800 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2b367e-e74c-4a65-92a7-ea170e652aae" containerName="registry" Sep 15 11:41:47.299012 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.298818 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2b367e-e74c-4a65-92a7-ea170e652aae" containerName="registry" Sep 15 11:41:47.299012 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.298900 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2b367e-e74c-4a65-92a7-ea170e652aae" containerName="registry" Sep 15 11:41:47.301666 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.301645 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.303942 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.303923 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cpjxd\"" Sep 15 11:41:47.304288 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.304257 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 15 11:41:47.304288 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.304276 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 15 11:41:47.304425 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.304307 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 15 11:41:47.304425 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.304324 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 15 11:41:47.320140 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.320114 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk"] Sep 15 11:41:47.457789 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.457755 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-tmp\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.457936 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.457796 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-manifests\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.457936 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.457826 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.457936 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.457888 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.457936 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.457905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtjbb\" (UniqueName: \"kubernetes.io/projected/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-kube-api-access-rtjbb\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.558992 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.558912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-manifests\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.558992 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.558964 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.559211 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.559003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.559211 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.559038 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtjbb\" (UniqueName: \"kubernetes.io/projected/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-kube-api-access-rtjbb\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.559211 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.559062 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-tmp\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.559395 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.559367 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-manifests\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.561292 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.561269 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-tmp\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.561535 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.561511 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.561594 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.561579 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.567470 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.567447 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtjbb\" (UniqueName: \"kubernetes.io/projected/47e85153-582d-4d5b-8c7e-1f56b2f6d0c8-kube-api-access-rtjbb\") pod \"opendatahub-operator-controller-manager-74bdb8b799-8nzlk\" (UID: \"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8\") " pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.611591 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.611558 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:47.895147 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:47.895123 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk"] Sep 15 11:41:47.898228 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:41:47.898204 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e85153_582d_4d5b_8c7e_1f56b2f6d0c8.slice/crio-8740bbafa354591f24b9e86f28bd75f428c4c97a7123c8ac900a70558d9be35b WatchSource:0}: Error finding container 8740bbafa354591f24b9e86f28bd75f428c4c97a7123c8ac900a70558d9be35b: Status 404 returned error can't find the container with id 8740bbafa354591f24b9e86f28bd75f428c4c97a7123c8ac900a70558d9be35b Sep 15 11:41:48.493170 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:48.493123 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" event={"ID":"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8","Type":"ContainerStarted","Data":"8740bbafa354591f24b9e86f28bd75f428c4c97a7123c8ac900a70558d9be35b"} Sep 15 11:41:51.509345 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:51.509314 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" event={"ID":"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8","Type":"ContainerStarted","Data":"57385e66ca565316be25e15bcd2d313d0bc565e6889e49fe99dfb52ac55da5f0"} Sep 15 11:41:52.514976 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:52.514938 2573 generic.go:356] "Generic (PLEG): container finished" podID="47e85153-582d-4d5b-8c7e-1f56b2f6d0c8" containerID="57385e66ca565316be25e15bcd2d313d0bc565e6889e49fe99dfb52ac55da5f0" exitCode=0 Sep 15 11:41:52.515368 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:52.515010 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" event={"ID":"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8","Type":"ContainerDied","Data":"57385e66ca565316be25e15bcd2d313d0bc565e6889e49fe99dfb52ac55da5f0"} Sep 15 11:41:53.174839 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:53.174811 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 11:41:53.177222 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:53.177188 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 11:41:53.186134 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:53.186103 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 11:41:53.187646 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:53.187622 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 11:41:53.189303 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:53.189281 2573 kubelet.go:1648] "Image garbage collection succeeded" Sep 15 11:41:53.521565 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:53.521526 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" event={"ID":"47e85153-582d-4d5b-8c7e-1f56b2f6d0c8","Type":"ContainerStarted","Data":"2325496ebf216cf8eaa373a6b548ae6e057b369fe85fb9b04b5916b9fd66e20d"} Sep 15 11:41:53.523996 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:53.521622 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:41:53.546189 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:41:53.546139 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" podStartSLOduration=1.9947612399999999 podStartE2EDuration="6.546123849s" totalImagesPullingTime="3.527433967s" totalInitContainerRuntime="1.023928642s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:41:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:41:53.544339114 +0000 UTC m=+300.774265650" watchObservedRunningTime="2026-09-15 11:41:53.546123849 +0000 UTC m=+300.776050386" Sep 15 11:42:04.528789 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:04.528760 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74bdb8b799-8nzlk" Sep 15 11:42:06.870400 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:06.870364 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-687d9c5d49-qph89"] Sep 15 11:42:06.879689 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:06.879662 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" Sep 15 11:42:06.882486 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:06.882463 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 15 11:42:06.882627 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:06.882555 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dk24m\"" Sep 15 11:42:06.883457 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:06.883439 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 15 11:42:06.886976 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:06.886955 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-687d9c5d49-qph89"] Sep 15 11:42:07.011212 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:07.011182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6afee05-aacb-4d71-b6f2-57122da1548a-tls-certs\") pod \"kube-auth-proxy-687d9c5d49-qph89\" (UID: \"f6afee05-aacb-4d71-b6f2-57122da1548a\") " pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" Sep 15 11:42:07.011380 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:07.011228 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h67b5\" (UniqueName: \"kubernetes.io/projected/f6afee05-aacb-4d71-b6f2-57122da1548a-kube-api-access-h67b5\") pod \"kube-auth-proxy-687d9c5d49-qph89\" (UID: \"f6afee05-aacb-4d71-b6f2-57122da1548a\") " pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" Sep 15 11:42:07.011380 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:07.011256 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6afee05-aacb-4d71-b6f2-57122da1548a-tmp\") pod \"kube-auth-proxy-687d9c5d49-qph89\" (UID: \"f6afee05-aacb-4d71-b6f2-57122da1548a\") " pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" Sep 15 11:42:07.111693 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:07.111659 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6afee05-aacb-4d71-b6f2-57122da1548a-tls-certs\") pod \"kube-auth-proxy-687d9c5d49-qph89\" (UID: \"f6afee05-aacb-4d71-b6f2-57122da1548a\") " pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" Sep 15 11:42:07.111879 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:07.111702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h67b5\" (UniqueName: \"kubernetes.io/projected/f6afee05-aacb-4d71-b6f2-57122da1548a-kube-api-access-h67b5\") pod \"kube-auth-proxy-687d9c5d49-qph89\" (UID: \"f6afee05-aacb-4d71-b6f2-57122da1548a\") " pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" Sep 15 11:42:07.111879 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:07.111724 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6afee05-aacb-4d71-b6f2-57122da1548a-tmp\") pod \"kube-auth-proxy-687d9c5d49-qph89\" (UID: \"f6afee05-aacb-4d71-b6f2-57122da1548a\") " pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" Sep 15 11:42:07.113983 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:07.113950 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6afee05-aacb-4d71-b6f2-57122da1548a-tmp\") pod \"kube-auth-proxy-687d9c5d49-qph89\" (UID: \"f6afee05-aacb-4d71-b6f2-57122da1548a\") " pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" Sep 15 11:42:07.114217 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:07.114193 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6afee05-aacb-4d71-b6f2-57122da1548a-tls-certs\") pod \"kube-auth-proxy-687d9c5d49-qph89\" (UID: \"f6afee05-aacb-4d71-b6f2-57122da1548a\") " pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" Sep 15 11:42:07.120773 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:07.120721 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h67b5\" (UniqueName: \"kubernetes.io/projected/f6afee05-aacb-4d71-b6f2-57122da1548a-kube-api-access-h67b5\") pod \"kube-auth-proxy-687d9c5d49-qph89\" (UID: \"f6afee05-aacb-4d71-b6f2-57122da1548a\") " pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" Sep 15 11:42:07.190117 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:07.190086 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" Sep 15 11:42:07.463079 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:07.462965 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-687d9c5d49-qph89"] Sep 15 11:42:07.465874 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:42:07.465849 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6afee05_aacb_4d71_b6f2_57122da1548a.slice/crio-3dc28dab0e6c869af09516605a54e8fade8dc8cfc82c7449e0aeb85ed3dd36c4 WatchSource:0}: Error finding container 3dc28dab0e6c869af09516605a54e8fade8dc8cfc82c7449e0aeb85ed3dd36c4: Status 404 returned error can't find the container with id 3dc28dab0e6c869af09516605a54e8fade8dc8cfc82c7449e0aeb85ed3dd36c4 Sep 15 11:42:07.468135 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:07.468116 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 11:42:07.580177 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:07.580136 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" event={"ID":"f6afee05-aacb-4d71-b6f2-57122da1548a","Type":"ContainerStarted","Data":"3dc28dab0e6c869af09516605a54e8fade8dc8cfc82c7449e0aeb85ed3dd36c4"} Sep 15 11:42:08.151548 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.151511 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w"] Sep 15 11:42:08.154225 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.154110 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.156949 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.156868 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 15 11:42:08.157153 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.156989 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 15 11:42:08.157153 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.157004 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-b6dls\"" Sep 15 11:42:08.157615 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.157410 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 15 11:42:08.157615 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.157431 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 15 11:42:08.166751 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.166692 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w"] Sep 15 11:42:08.323377 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.323340 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvgj9\" (UniqueName: \"kubernetes.io/projected/9a8566ee-d876-4722-8cf5-3172e4f6c94e-kube-api-access-jvgj9\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.323377 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.323388 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.323630 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.323413 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.323630 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.323468 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.323630 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.323532 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8566ee-d876-4722-8cf5-3172e4f6c94e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.323630 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.323604 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.323838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.323634 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.424942 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.424786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgj9\" (UniqueName: \"kubernetes.io/projected/9a8566ee-d876-4722-8cf5-3172e4f6c94e-kube-api-access-jvgj9\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.424942 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.424843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.424942 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.424880 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.424942 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.424905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.425310 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.424957 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8566ee-d876-4722-8cf5-3172e4f6c94e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.425310 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.425034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.425310 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.425061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.425919 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.425859 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.428679 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.428632 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.428765 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.428678 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8566ee-d876-4722-8cf5-3172e4f6c94e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.428986 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.428960 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.429472 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.429431 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.433470 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.433423 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.434473 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.434449 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgj9\" (UniqueName: \"kubernetes.io/projected/9a8566ee-d876-4722-8cf5-3172e4f6c94e-kube-api-access-jvgj9\") pod \"istiod-openshift-gateway-7b7995d5cd-zwv8w\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.469073 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.469047 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:08.752544 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:08.752516 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w"] Sep 15 11:42:08.755067 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:42:08.755017 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8566ee_d876_4722_8cf5_3172e4f6c94e.slice/crio-a033283d2325cd071934f49bca74a381d1fdf85d44ad81bec7024eba533e6b78 WatchSource:0}: Error finding container a033283d2325cd071934f49bca74a381d1fdf85d44ad81bec7024eba533e6b78: Status 404 returned error can't find the container with id a033283d2325cd071934f49bca74a381d1fdf85d44ad81bec7024eba533e6b78 Sep 15 11:42:09.590972 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:09.590933 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" event={"ID":"9a8566ee-d876-4722-8cf5-3172e4f6c94e","Type":"ContainerStarted","Data":"a033283d2325cd071934f49bca74a381d1fdf85d44ad81bec7024eba533e6b78"} Sep 15 11:42:10.599951 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:10.599674 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" event={"ID":"f6afee05-aacb-4d71-b6f2-57122da1548a","Type":"ContainerStarted","Data":"395d1f693f110691d923adf4b6aaf9c6629a2395d3fa08076160ee2d5617b4f8"} Sep 15 11:42:10.623580 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:10.623460 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-687d9c5d49-qph89" podStartSLOduration=1.9256280380000002 podStartE2EDuration="4.62344086s" totalImagesPullingTime="2.697812822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:42:10.621848492 +0000 UTC m=+317.851775029" watchObservedRunningTime="2026-09-15 11:42:10.62344086 +0000 UTC m=+317.853367409" Sep 15 11:42:12.142790 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:12.142746 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 11:42:12.143013 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:12.142846 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 15 11:42:12.610970 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:12.610932 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" event={"ID":"9a8566ee-d876-4722-8cf5-3172e4f6c94e","Type":"ContainerStarted","Data":"835679a58d46aae17bda44e15775f6590822b36ca33b1157e2268ebc94ab4ca6"} Sep 15 11:42:12.611163 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:12.611067 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:13.616801 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:13.616763 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-zwv8w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 11:42:13.617167 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:13.616841 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" podUID="9a8566ee-d876-4722-8cf5-3172e4f6c94e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 11:42:16.616083 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:16.616048 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:16.637692 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:16.637647 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" podStartSLOduration=5.253373097 podStartE2EDuration="8.637609491s" totalImagesPullingTime="3.384236394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:42:12.650246708 +0000 UTC m=+319.880173246" watchObservedRunningTime="2026-09-15 11:42:16.637609491 +0000 UTC m=+323.867536027" Sep 15 11:42:24.989007 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:24.988972 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd"] Sep 15 11:42:24.993641 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:24.993621 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:24.997458 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:24.997434 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 15 11:42:24.997578 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:24.997466 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 15 11:42:24.997578 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:24.997467 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 15 11:42:24.997578 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:24.997466 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 15 11:42:24.997578 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:24.997557 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 15 11:42:24.997578 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:24.997445 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-h2jx9\"" Sep 15 11:42:25.000972 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.000949 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd"] Sep 15 11:42:25.053165 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.053135 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2417650e-463e-429f-be11-127663b61d01-manager-config\") pod \"lws-controller-manager-8487688d7-g6mdd\" (UID: \"2417650e-463e-429f-be11-127663b61d01\") " pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:25.053334 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.053177 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2417650e-463e-429f-be11-127663b61d01-metrics-cert\") pod \"lws-controller-manager-8487688d7-g6mdd\" (UID: \"2417650e-463e-429f-be11-127663b61d01\") " pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:25.053334 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.053196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5fs\" (UniqueName: \"kubernetes.io/projected/2417650e-463e-429f-be11-127663b61d01-kube-api-access-qj5fs\") pod \"lws-controller-manager-8487688d7-g6mdd\" (UID: \"2417650e-463e-429f-be11-127663b61d01\") " pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:25.053334 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.053220 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2417650e-463e-429f-be11-127663b61d01-cert\") pod \"lws-controller-manager-8487688d7-g6mdd\" (UID: \"2417650e-463e-429f-be11-127663b61d01\") " pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:25.154269 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.154230 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2417650e-463e-429f-be11-127663b61d01-cert\") pod \"lws-controller-manager-8487688d7-g6mdd\" (UID: \"2417650e-463e-429f-be11-127663b61d01\") " pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:25.154459 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.154313 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2417650e-463e-429f-be11-127663b61d01-manager-config\") pod \"lws-controller-manager-8487688d7-g6mdd\" (UID: \"2417650e-463e-429f-be11-127663b61d01\") " pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:25.154459 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.154361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2417650e-463e-429f-be11-127663b61d01-metrics-cert\") pod \"lws-controller-manager-8487688d7-g6mdd\" (UID: \"2417650e-463e-429f-be11-127663b61d01\") " pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:25.154459 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.154385 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5fs\" (UniqueName: \"kubernetes.io/projected/2417650e-463e-429f-be11-127663b61d01-kube-api-access-qj5fs\") pod \"lws-controller-manager-8487688d7-g6mdd\" (UID: \"2417650e-463e-429f-be11-127663b61d01\") " pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:25.155074 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.155047 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2417650e-463e-429f-be11-127663b61d01-manager-config\") pod \"lws-controller-manager-8487688d7-g6mdd\" (UID: \"2417650e-463e-429f-be11-127663b61d01\") " pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:25.156627 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.156608 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2417650e-463e-429f-be11-127663b61d01-metrics-cert\") pod \"lws-controller-manager-8487688d7-g6mdd\" (UID: \"2417650e-463e-429f-be11-127663b61d01\") " pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:25.156737 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.156719 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2417650e-463e-429f-be11-127663b61d01-cert\") pod \"lws-controller-manager-8487688d7-g6mdd\" (UID: \"2417650e-463e-429f-be11-127663b61d01\") " pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:25.174656 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.174623 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5fs\" (UniqueName: \"kubernetes.io/projected/2417650e-463e-429f-be11-127663b61d01-kube-api-access-qj5fs\") pod \"lws-controller-manager-8487688d7-g6mdd\" (UID: \"2417650e-463e-429f-be11-127663b61d01\") " pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:25.302837 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.302799 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:25.595090 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.595065 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd"] Sep 15 11:42:25.597199 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:42:25.597165 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2417650e_463e_429f_be11_127663b61d01.slice/crio-3043f061ae94f6737c464f9afb08d7af1d42e12f53037b49a2b061ba437ecd14 WatchSource:0}: Error finding container 3043f061ae94f6737c464f9afb08d7af1d42e12f53037b49a2b061ba437ecd14: Status 404 returned error can't find the container with id 3043f061ae94f6737c464f9afb08d7af1d42e12f53037b49a2b061ba437ecd14 Sep 15 11:42:25.666348 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:25.666317 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" event={"ID":"2417650e-463e-429f-be11-127663b61d01","Type":"ContainerStarted","Data":"3043f061ae94f6737c464f9afb08d7af1d42e12f53037b49a2b061ba437ecd14"} Sep 15 11:42:28.292381 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.292345 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w"] Sep 15 11:42:28.292754 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.292521 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" podUID="9a8566ee-d876-4722-8cf5-3172e4f6c94e" containerName="discovery" containerID="cri-o://835679a58d46aae17bda44e15775f6590822b36ca33b1157e2268ebc94ab4ca6" gracePeriod=30 Sep 15 11:42:28.524786 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.524761 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:28.585700 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.585668 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-csr-ca-configmap\") pod \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " Sep 15 11:42:28.585814 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.585714 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-csr-dns-cert\") pod \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " Sep 15 11:42:28.585814 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.585762 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvgj9\" (UniqueName: \"kubernetes.io/projected/9a8566ee-d876-4722-8cf5-3172e4f6c94e-kube-api-access-jvgj9\") pod \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " Sep 15 11:42:28.585814 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.585798 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-cacerts\") pod \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " Sep 15 11:42:28.585936 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.585824 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-token\") pod \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " Sep 15 11:42:28.585936 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.585887 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8566ee-d876-4722-8cf5-3172e4f6c94e-local-certs\") pod \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " Sep 15 11:42:28.585936 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.585910 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-kubeconfig\") pod \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\" (UID: \"9a8566ee-d876-4722-8cf5-3172e4f6c94e\") " Sep 15 11:42:28.586173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.586145 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9a8566ee-d876-4722-8cf5-3172e4f6c94e" (UID: "9a8566ee-d876-4722-8cf5-3172e4f6c94e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:42:28.588141 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.588109 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9a8566ee-d876-4722-8cf5-3172e4f6c94e" (UID: "9a8566ee-d876-4722-8cf5-3172e4f6c94e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:42:28.588250 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.588113 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a8566ee-d876-4722-8cf5-3172e4f6c94e" (UID: "9a8566ee-d876-4722-8cf5-3172e4f6c94e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:42:28.588432 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.588408 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8566ee-d876-4722-8cf5-3172e4f6c94e-local-certs" (OuterVolumeSpecName: "local-certs") pod "9a8566ee-d876-4722-8cf5-3172e4f6c94e" (UID: "9a8566ee-d876-4722-8cf5-3172e4f6c94e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:42:28.588432 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.588421 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-cacerts" (OuterVolumeSpecName: "cacerts") pod "9a8566ee-d876-4722-8cf5-3172e4f6c94e" (UID: "9a8566ee-d876-4722-8cf5-3172e4f6c94e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:42:28.588570 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.588505 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9a8566ee-d876-4722-8cf5-3172e4f6c94e" (UID: "9a8566ee-d876-4722-8cf5-3172e4f6c94e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:42:28.588570 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.588523 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8566ee-d876-4722-8cf5-3172e4f6c94e-kube-api-access-jvgj9" (OuterVolumeSpecName: "kube-api-access-jvgj9") pod "9a8566ee-d876-4722-8cf5-3172e4f6c94e" (UID: "9a8566ee-d876-4722-8cf5-3172e4f6c94e"). InnerVolumeSpecName "kube-api-access-jvgj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:42:28.681650 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.681580 2573 generic.go:356] "Generic (PLEG): container finished" podID="9a8566ee-d876-4722-8cf5-3172e4f6c94e" containerID="835679a58d46aae17bda44e15775f6590822b36ca33b1157e2268ebc94ab4ca6" exitCode=0 Sep 15 11:42:28.681650 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.681632 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" Sep 15 11:42:28.681826 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.681669 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" event={"ID":"9a8566ee-d876-4722-8cf5-3172e4f6c94e","Type":"ContainerDied","Data":"835679a58d46aae17bda44e15775f6590822b36ca33b1157e2268ebc94ab4ca6"} Sep 15 11:42:28.681826 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.681734 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w" event={"ID":"9a8566ee-d876-4722-8cf5-3172e4f6c94e","Type":"ContainerDied","Data":"a033283d2325cd071934f49bca74a381d1fdf85d44ad81bec7024eba533e6b78"} Sep 15 11:42:28.681826 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.681755 2573 scope.go:117] "RemoveContainer" containerID="835679a58d46aae17bda44e15775f6590822b36ca33b1157e2268ebc94ab4ca6" Sep 15 11:42:28.683391 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.683364 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" event={"ID":"2417650e-463e-429f-be11-127663b61d01","Type":"ContainerStarted","Data":"7737052e3eda1b2912a7b8c219f12967244da7cdd07a5d7cbe257eb3f49e0d4e"} Sep 15 11:42:28.683487 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.683468 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:42:28.687273 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.687252 2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-csr-ca-configmap\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:42:28.687273 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.687273 2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-csr-dns-cert\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:42:28.687385 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.687283 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvgj9\" (UniqueName: \"kubernetes.io/projected/9a8566ee-d876-4722-8cf5-3172e4f6c94e-kube-api-access-jvgj9\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:42:28.687385 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.687292 2573 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-cacerts\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:42:28.687385 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.687301 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-token\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:42:28.687385 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.687309 2573 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8566ee-d876-4722-8cf5-3172e4f6c94e-local-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:42:28.687385 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.687317 2573 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a8566ee-d876-4722-8cf5-3172e4f6c94e-istio-kubeconfig\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:42:28.690383 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.690364 2573 scope.go:117] "RemoveContainer" containerID="835679a58d46aae17bda44e15775f6590822b36ca33b1157e2268ebc94ab4ca6" Sep 15 11:42:28.692282 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:42:28.690639 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835679a58d46aae17bda44e15775f6590822b36ca33b1157e2268ebc94ab4ca6\": container with ID starting with 835679a58d46aae17bda44e15775f6590822b36ca33b1157e2268ebc94ab4ca6 not found: ID does not exist" containerID="835679a58d46aae17bda44e15775f6590822b36ca33b1157e2268ebc94ab4ca6" Sep 15 11:42:28.692282 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.691068 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835679a58d46aae17bda44e15775f6590822b36ca33b1157e2268ebc94ab4ca6"} err="failed to get container status \"835679a58d46aae17bda44e15775f6590822b36ca33b1157e2268ebc94ab4ca6\": rpc error: code = NotFound desc = could not find container \"835679a58d46aae17bda44e15775f6590822b36ca33b1157e2268ebc94ab4ca6\": container with ID starting with 835679a58d46aae17bda44e15775f6590822b36ca33b1157e2268ebc94ab4ca6 not found: ID does not exist" Sep 15 11:42:28.723031 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.722987 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" podStartSLOduration=2.232679302 podStartE2EDuration="4.722973327s" totalImagesPullingTime="2.490294025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:42:28.721874632 +0000 UTC m=+335.951801182" watchObservedRunningTime="2026-09-15 11:42:28.722973327 +0000 UTC m=+335.952899922" Sep 15 11:42:28.746011 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.745983 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w"] Sep 15 11:42:28.748785 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:28.748761 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zwv8w"] Sep 15 11:42:29.234064 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:29.234012 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a8566ee-d876-4722-8cf5-3172e4f6c94e" path="/var/lib/kubelet/pods/9a8566ee-d876-4722-8cf5-3172e4f6c94e/volumes" Sep 15 11:42:39.690481 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:42:39.690448 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8487688d7-g6mdd" Sep 15 11:43:43.124188 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.124154 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p"] Sep 15 11:43:43.124582 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.124471 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8566ee-d876-4722-8cf5-3172e4f6c94e" containerName="discovery" Sep 15 11:43:43.124582 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.124483 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8566ee-d876-4722-8cf5-3172e4f6c94e" containerName="discovery" Sep 15 11:43:43.124582 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.124548 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8566ee-d876-4722-8cf5-3172e4f6c94e" containerName="discovery" Sep 15 11:43:43.127451 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.127434 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" Sep 15 11:43:43.129682 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.129657 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 15 11:43:43.130593 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.130573 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hhdjf\"" Sep 15 11:43:43.130710 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.130626 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 15 11:43:43.130710 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.130634 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 15 11:43:43.130710 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.130689 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 15 11:43:43.136568 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.136545 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p"] Sep 15 11:43:43.231151 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.231123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/51b2effb-31d6-4b96-b9a0-c9801353cb00-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9vw8p\" (UID: \"51b2effb-31d6-4b96-b9a0-c9801353cb00\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" Sep 15 11:43:43.231278 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.231167 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/51b2effb-31d6-4b96-b9a0-c9801353cb00-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9vw8p\" (UID: \"51b2effb-31d6-4b96-b9a0-c9801353cb00\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" Sep 15 11:43:43.231278 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.231246 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6759n\" (UniqueName: \"kubernetes.io/projected/51b2effb-31d6-4b96-b9a0-c9801353cb00-kube-api-access-6759n\") pod \"kuadrant-console-plugin-fbdd4f445-9vw8p\" (UID: \"51b2effb-31d6-4b96-b9a0-c9801353cb00\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" Sep 15 11:43:43.332196 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.332162 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/51b2effb-31d6-4b96-b9a0-c9801353cb00-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9vw8p\" (UID: \"51b2effb-31d6-4b96-b9a0-c9801353cb00\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" Sep 15 11:43:43.332354 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.332212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/51b2effb-31d6-4b96-b9a0-c9801353cb00-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9vw8p\" (UID: \"51b2effb-31d6-4b96-b9a0-c9801353cb00\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" Sep 15 11:43:43.332354 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.332252 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6759n\" (UniqueName: \"kubernetes.io/projected/51b2effb-31d6-4b96-b9a0-c9801353cb00-kube-api-access-6759n\") pod \"kuadrant-console-plugin-fbdd4f445-9vw8p\" (UID: \"51b2effb-31d6-4b96-b9a0-c9801353cb00\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" Sep 15 11:43:43.332354 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:43:43.332307 2573 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 15 11:43:43.332473 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:43:43.332403 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b2effb-31d6-4b96-b9a0-c9801353cb00-plugin-serving-cert podName:51b2effb-31d6-4b96-b9a0-c9801353cb00 nodeName:}" failed. No retries permitted until 2026-09-15 11:43:43.832380869 +0000 UTC m=+411.062307385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/51b2effb-31d6-4b96-b9a0-c9801353cb00-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-9vw8p" (UID: "51b2effb-31d6-4b96-b9a0-c9801353cb00") : secret "plugin-serving-cert" not found Sep 15 11:43:43.332945 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.332926 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/51b2effb-31d6-4b96-b9a0-c9801353cb00-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9vw8p\" (UID: \"51b2effb-31d6-4b96-b9a0-c9801353cb00\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" Sep 15 11:43:43.341983 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.341960 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6759n\" (UniqueName: \"kubernetes.io/projected/51b2effb-31d6-4b96-b9a0-c9801353cb00-kube-api-access-6759n\") pod \"kuadrant-console-plugin-fbdd4f445-9vw8p\" (UID: \"51b2effb-31d6-4b96-b9a0-c9801353cb00\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" Sep 15 11:43:43.835952 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.835916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/51b2effb-31d6-4b96-b9a0-c9801353cb00-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9vw8p\" (UID: \"51b2effb-31d6-4b96-b9a0-c9801353cb00\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" Sep 15 11:43:43.838282 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:43.838263 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/51b2effb-31d6-4b96-b9a0-c9801353cb00-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9vw8p\" (UID: \"51b2effb-31d6-4b96-b9a0-c9801353cb00\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" Sep 15 11:43:44.037163 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:44.037128 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" Sep 15 11:43:44.308941 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:44.308904 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p"] Sep 15 11:43:44.312136 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:43:44.312108 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51b2effb_31d6_4b96_b9a0_c9801353cb00.slice/crio-790f8e7f7d413a06c56d68b8c7845482f3c4ebce7aaa41e296d229e82e10bd91 WatchSource:0}: Error finding container 790f8e7f7d413a06c56d68b8c7845482f3c4ebce7aaa41e296d229e82e10bd91: Status 404 returned error can't find the container with id 790f8e7f7d413a06c56d68b8c7845482f3c4ebce7aaa41e296d229e82e10bd91 Sep 15 11:43:44.993926 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:44.993876 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" event={"ID":"51b2effb-31d6-4b96-b9a0-c9801353cb00","Type":"ContainerStarted","Data":"790f8e7f7d413a06c56d68b8c7845482f3c4ebce7aaa41e296d229e82e10bd91"} Sep 15 11:43:49.018353 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:49.018316 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" event={"ID":"51b2effb-31d6-4b96-b9a0-c9801353cb00","Type":"ContainerStarted","Data":"d468251dbc0c6f3b3b2feefc0cbecee547fad5b52dc3f6a1ad0050234ae5f7af"} Sep 15 11:43:49.034300 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:43:49.034246 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9vw8p" podStartSLOduration=1.6107970200000001 podStartE2EDuration="6.034230564s" totalImagesPullingTime="4.423433544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:43:49.032991003 +0000 UTC m=+416.262917551" watchObservedRunningTime="2026-09-15 11:43:49.034230564 +0000 UTC m=+416.264157100" Sep 15 11:44:32.345197 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.345158 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cfd25"] Sep 15 11:44:32.348521 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.348500 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cfd25" Sep 15 11:44:32.350537 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.350520 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4289k\"" Sep 15 11:44:32.357459 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.357437 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cfd25"] Sep 15 11:44:32.431226 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.431196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b82jd\" (UniqueName: \"kubernetes.io/projected/43477c56-aefe-4b2f-8b64-d1cb1c3cd91c-kube-api-access-b82jd\") pod \"authorino-7bcfd8f9cd-cfd25\" (UID: \"43477c56-aefe-4b2f-8b64-d1cb1c3cd91c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-cfd25" Sep 15 11:44:32.466077 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.466045 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-sdmth"] Sep 15 11:44:32.469015 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.468999 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sdmth" Sep 15 11:44:32.476986 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.476966 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sdmth"] Sep 15 11:44:32.532463 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.532433 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b82jd\" (UniqueName: \"kubernetes.io/projected/43477c56-aefe-4b2f-8b64-d1cb1c3cd91c-kube-api-access-b82jd\") pod \"authorino-7bcfd8f9cd-cfd25\" (UID: \"43477c56-aefe-4b2f-8b64-d1cb1c3cd91c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-cfd25" Sep 15 11:44:32.532618 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.532501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfhq7\" (UniqueName: \"kubernetes.io/projected/5b3787cb-fa22-4fdd-84a4-63810e5abd03-kube-api-access-zfhq7\") pod \"authorino-59c974bd48-sdmth\" (UID: \"5b3787cb-fa22-4fdd-84a4-63810e5abd03\") " pod="kuadrant-system/authorino-59c974bd48-sdmth" Sep 15 11:44:32.540039 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.539995 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b82jd\" (UniqueName: \"kubernetes.io/projected/43477c56-aefe-4b2f-8b64-d1cb1c3cd91c-kube-api-access-b82jd\") pod \"authorino-7bcfd8f9cd-cfd25\" (UID: \"43477c56-aefe-4b2f-8b64-d1cb1c3cd91c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-cfd25" Sep 15 11:44:32.632894 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.632811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfhq7\" (UniqueName: \"kubernetes.io/projected/5b3787cb-fa22-4fdd-84a4-63810e5abd03-kube-api-access-zfhq7\") pod \"authorino-59c974bd48-sdmth\" (UID: \"5b3787cb-fa22-4fdd-84a4-63810e5abd03\") " pod="kuadrant-system/authorino-59c974bd48-sdmth" Sep 15 11:44:32.641009 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.640983 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfhq7\" (UniqueName: \"kubernetes.io/projected/5b3787cb-fa22-4fdd-84a4-63810e5abd03-kube-api-access-zfhq7\") pod \"authorino-59c974bd48-sdmth\" (UID: \"5b3787cb-fa22-4fdd-84a4-63810e5abd03\") " pod="kuadrant-system/authorino-59c974bd48-sdmth" Sep 15 11:44:32.658163 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.658139 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cfd25" Sep 15 11:44:32.777674 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.777640 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sdmth" Sep 15 11:44:32.924128 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:32.924100 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cfd25"] Sep 15 11:44:32.926629 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:44:32.926601 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43477c56_aefe_4b2f_8b64_d1cb1c3cd91c.slice/crio-e17cf2108ac5a3445199c138d6fdb6d95a93203fdef1596cd0d4d4c258554e95 WatchSource:0}: Error finding container e17cf2108ac5a3445199c138d6fdb6d95a93203fdef1596cd0d4d4c258554e95: Status 404 returned error can't find the container with id e17cf2108ac5a3445199c138d6fdb6d95a93203fdef1596cd0d4d4c258554e95 Sep 15 11:44:33.046056 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:33.046010 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sdmth"] Sep 15 11:44:33.049467 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:44:33.049441 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b3787cb_fa22_4fdd_84a4_63810e5abd03.slice/crio-b1db63194c3cc5ed62d730d0cdbc87289499f309ec07bf0efcca90ca41845318 WatchSource:0}: Error finding container b1db63194c3cc5ed62d730d0cdbc87289499f309ec07bf0efcca90ca41845318: Status 404 returned error can't find the container with id b1db63194c3cc5ed62d730d0cdbc87289499f309ec07bf0efcca90ca41845318 Sep 15 11:44:33.195866 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:33.195784 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sdmth" event={"ID":"5b3787cb-fa22-4fdd-84a4-63810e5abd03","Type":"ContainerStarted","Data":"b1db63194c3cc5ed62d730d0cdbc87289499f309ec07bf0efcca90ca41845318"} Sep 15 11:44:33.196886 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:33.196860 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cfd25" event={"ID":"43477c56-aefe-4b2f-8b64-d1cb1c3cd91c","Type":"ContainerStarted","Data":"e17cf2108ac5a3445199c138d6fdb6d95a93203fdef1596cd0d4d4c258554e95"} Sep 15 11:44:36.212978 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:36.212942 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sdmth" event={"ID":"5b3787cb-fa22-4fdd-84a4-63810e5abd03","Type":"ContainerStarted","Data":"c375902ad8ee6c7f92996013f72bcf135d43275f9c782a6772f998b466f9e719"} Sep 15 11:44:36.214501 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:36.214470 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cfd25" event={"ID":"43477c56-aefe-4b2f-8b64-d1cb1c3cd91c","Type":"ContainerStarted","Data":"719acc4d65f7410bd338a29aaaae576f6f1b4a7dcb4def0f46ded93a73579335"} Sep 15 11:44:36.229157 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:36.229112 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-sdmth" podStartSLOduration=1.8355861980000001 podStartE2EDuration="4.229099655s" totalImagesPullingTime="2.393513457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:44:36.22814848 +0000 UTC m=+463.458075016" watchObservedRunningTime="2026-09-15 11:44:36.229099655 +0000 UTC m=+463.459026184" Sep 15 11:44:36.243184 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:36.243138 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-cfd25" podStartSLOduration=1.727190841 podStartE2EDuration="4.243124092s" totalImagesPullingTime="2.515933251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:44:36.241746458 +0000 UTC m=+463.471672997" watchObservedRunningTime="2026-09-15 11:44:36.243124092 +0000 UTC m=+463.473050628" Sep 15 11:44:36.266331 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:36.266304 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cfd25"] Sep 15 11:44:38.222347 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:38.222306 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-cfd25" podUID="43477c56-aefe-4b2f-8b64-d1cb1c3cd91c" containerName="authorino" containerID="cri-o://719acc4d65f7410bd338a29aaaae576f6f1b4a7dcb4def0f46ded93a73579335" gracePeriod=30 Sep 15 11:44:38.464247 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:38.464223 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cfd25" Sep 15 11:44:38.480942 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:38.480882 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b82jd\" (UniqueName: \"kubernetes.io/projected/43477c56-aefe-4b2f-8b64-d1cb1c3cd91c-kube-api-access-b82jd\") pod \"43477c56-aefe-4b2f-8b64-d1cb1c3cd91c\" (UID: \"43477c56-aefe-4b2f-8b64-d1cb1c3cd91c\") " Sep 15 11:44:38.483132 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:38.483101 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43477c56-aefe-4b2f-8b64-d1cb1c3cd91c-kube-api-access-b82jd" (OuterVolumeSpecName: "kube-api-access-b82jd") pod "43477c56-aefe-4b2f-8b64-d1cb1c3cd91c" (UID: "43477c56-aefe-4b2f-8b64-d1cb1c3cd91c"). InnerVolumeSpecName "kube-api-access-b82jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:44:38.581722 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:38.581676 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b82jd\" (UniqueName: \"kubernetes.io/projected/43477c56-aefe-4b2f-8b64-d1cb1c3cd91c-kube-api-access-b82jd\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:44:39.227354 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:39.227321 2573 generic.go:356] "Generic (PLEG): container finished" podID="43477c56-aefe-4b2f-8b64-d1cb1c3cd91c" containerID="719acc4d65f7410bd338a29aaaae576f6f1b4a7dcb4def0f46ded93a73579335" exitCode=0 Sep 15 11:44:39.227735 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:39.227360 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cfd25" Sep 15 11:44:39.227735 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:39.227387 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cfd25" event={"ID":"43477c56-aefe-4b2f-8b64-d1cb1c3cd91c","Type":"ContainerDied","Data":"719acc4d65f7410bd338a29aaaae576f6f1b4a7dcb4def0f46ded93a73579335"} Sep 15 11:44:39.227735 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:39.227418 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cfd25" event={"ID":"43477c56-aefe-4b2f-8b64-d1cb1c3cd91c","Type":"ContainerDied","Data":"e17cf2108ac5a3445199c138d6fdb6d95a93203fdef1596cd0d4d4c258554e95"} Sep 15 11:44:39.227735 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:39.227436 2573 scope.go:117] "RemoveContainer" containerID="719acc4d65f7410bd338a29aaaae576f6f1b4a7dcb4def0f46ded93a73579335" Sep 15 11:44:39.235525 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:39.235505 2573 scope.go:117] "RemoveContainer" containerID="719acc4d65f7410bd338a29aaaae576f6f1b4a7dcb4def0f46ded93a73579335" Sep 15 11:44:39.235868 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:44:39.235830 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719acc4d65f7410bd338a29aaaae576f6f1b4a7dcb4def0f46ded93a73579335\": container with ID starting with 719acc4d65f7410bd338a29aaaae576f6f1b4a7dcb4def0f46ded93a73579335 not found: ID does not exist" containerID="719acc4d65f7410bd338a29aaaae576f6f1b4a7dcb4def0f46ded93a73579335" Sep 15 11:44:39.235936 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:39.235863 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719acc4d65f7410bd338a29aaaae576f6f1b4a7dcb4def0f46ded93a73579335"} err="failed to get container status \"719acc4d65f7410bd338a29aaaae576f6f1b4a7dcb4def0f46ded93a73579335\": rpc error: code = NotFound desc = could not find container \"719acc4d65f7410bd338a29aaaae576f6f1b4a7dcb4def0f46ded93a73579335\": container with ID starting with 719acc4d65f7410bd338a29aaaae576f6f1b4a7dcb4def0f46ded93a73579335 not found: ID does not exist" Sep 15 11:44:39.248728 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:39.248707 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cfd25"] Sep 15 11:44:39.252844 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:39.252821 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cfd25"] Sep 15 11:44:41.234594 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:44:41.234552 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43477c56-aefe-4b2f-8b64-d1cb1c3cd91c" path="/var/lib/kubelet/pods/43477c56-aefe-4b2f-8b64-d1cb1c3cd91c/volumes" Sep 15 11:45:01.462691 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.462657 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54766dc64c-4pkgn"] Sep 15 11:45:01.463059 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.462990 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43477c56-aefe-4b2f-8b64-d1cb1c3cd91c" containerName="authorino" Sep 15 11:45:01.463059 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.463003 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="43477c56-aefe-4b2f-8b64-d1cb1c3cd91c" containerName="authorino" Sep 15 11:45:01.463142 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.463088 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="43477c56-aefe-4b2f-8b64-d1cb1c3cd91c" containerName="authorino" Sep 15 11:45:01.467135 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.467110 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54766dc64c-4pkgn" Sep 15 11:45:01.486656 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.486621 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54766dc64c-4pkgn"] Sep 15 11:45:01.561916 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.561880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffsn\" (UniqueName: \"kubernetes.io/projected/2d432c92-1211-4a42-880d-af9132134113-kube-api-access-nffsn\") pod \"authorino-54766dc64c-4pkgn\" (UID: \"2d432c92-1211-4a42-880d-af9132134113\") " pod="kuadrant-system/authorino-54766dc64c-4pkgn" Sep 15 11:45:01.650152 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.650120 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54766dc64c-4pkgn"] Sep 15 11:45:01.650367 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:45:01.650346 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nffsn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-54766dc64c-4pkgn" podUID="2d432c92-1211-4a42-880d-af9132134113" Sep 15 11:45:01.662797 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.662769 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nffsn\" (UniqueName: \"kubernetes.io/projected/2d432c92-1211-4a42-880d-af9132134113-kube-api-access-nffsn\") pod \"authorino-54766dc64c-4pkgn\" (UID: \"2d432c92-1211-4a42-880d-af9132134113\") " pod="kuadrant-system/authorino-54766dc64c-4pkgn" Sep 15 11:45:01.674110 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.674085 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffsn\" (UniqueName: \"kubernetes.io/projected/2d432c92-1211-4a42-880d-af9132134113-kube-api-access-nffsn\") pod \"authorino-54766dc64c-4pkgn\" (UID: \"2d432c92-1211-4a42-880d-af9132134113\") " pod="kuadrant-system/authorino-54766dc64c-4pkgn" Sep 15 11:45:01.685412 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.685389 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fc7f4c957-skzpv"] Sep 15 11:45:01.689360 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.689124 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc7f4c957-skzpv" Sep 15 11:45:01.691481 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.691456 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 15 11:45:01.698894 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.698870 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fc7f4c957-skzpv"] Sep 15 11:45:01.764124 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.764089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2sj\" (UniqueName: \"kubernetes.io/projected/517f5bf5-39d6-4c7e-a902-717d325bdf77-kube-api-access-wh2sj\") pod \"authorino-5fc7f4c957-skzpv\" (UID: \"517f5bf5-39d6-4c7e-a902-717d325bdf77\") " pod="kuadrant-system/authorino-5fc7f4c957-skzpv" Sep 15 11:45:01.764318 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.764142 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/517f5bf5-39d6-4c7e-a902-717d325bdf77-tls-cert\") pod \"authorino-5fc7f4c957-skzpv\" (UID: \"517f5bf5-39d6-4c7e-a902-717d325bdf77\") " pod="kuadrant-system/authorino-5fc7f4c957-skzpv" Sep 15 11:45:01.864918 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.864877 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2sj\" (UniqueName: \"kubernetes.io/projected/517f5bf5-39d6-4c7e-a902-717d325bdf77-kube-api-access-wh2sj\") pod \"authorino-5fc7f4c957-skzpv\" (UID: \"517f5bf5-39d6-4c7e-a902-717d325bdf77\") " pod="kuadrant-system/authorino-5fc7f4c957-skzpv" Sep 15 11:45:01.865135 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.864931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/517f5bf5-39d6-4c7e-a902-717d325bdf77-tls-cert\") pod \"authorino-5fc7f4c957-skzpv\" (UID: \"517f5bf5-39d6-4c7e-a902-717d325bdf77\") " pod="kuadrant-system/authorino-5fc7f4c957-skzpv" Sep 15 11:45:01.867346 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.867317 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/517f5bf5-39d6-4c7e-a902-717d325bdf77-tls-cert\") pod \"authorino-5fc7f4c957-skzpv\" (UID: \"517f5bf5-39d6-4c7e-a902-717d325bdf77\") " pod="kuadrant-system/authorino-5fc7f4c957-skzpv" Sep 15 11:45:01.874059 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:01.874010 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2sj\" (UniqueName: \"kubernetes.io/projected/517f5bf5-39d6-4c7e-a902-717d325bdf77-kube-api-access-wh2sj\") pod \"authorino-5fc7f4c957-skzpv\" (UID: \"517f5bf5-39d6-4c7e-a902-717d325bdf77\") " pod="kuadrant-system/authorino-5fc7f4c957-skzpv" Sep 15 11:45:02.000803 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:02.000769 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc7f4c957-skzpv" Sep 15 11:45:02.271244 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:02.271218 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fc7f4c957-skzpv"] Sep 15 11:45:02.273184 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:45:02.273154 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod517f5bf5_39d6_4c7e_a902_717d325bdf77.slice/crio-a039e98fc1c2b0b785dba9c68851b91fe6f3991a4b2f4af286989d3d3883304b WatchSource:0}: Error finding container a039e98fc1c2b0b785dba9c68851b91fe6f3991a4b2f4af286989d3d3883304b: Status 404 returned error can't find the container with id a039e98fc1c2b0b785dba9c68851b91fe6f3991a4b2f4af286989d3d3883304b Sep 15 11:45:02.324250 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:02.324177 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54766dc64c-4pkgn" Sep 15 11:45:02.324250 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:02.324189 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc7f4c957-skzpv" event={"ID":"517f5bf5-39d6-4c7e-a902-717d325bdf77","Type":"ContainerStarted","Data":"a039e98fc1c2b0b785dba9c68851b91fe6f3991a4b2f4af286989d3d3883304b"} Sep 15 11:45:02.332101 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:02.332081 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54766dc64c-4pkgn" Sep 15 11:45:02.469655 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:02.469618 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nffsn\" (UniqueName: \"kubernetes.io/projected/2d432c92-1211-4a42-880d-af9132134113-kube-api-access-nffsn\") pod \"2d432c92-1211-4a42-880d-af9132134113\" (UID: \"2d432c92-1211-4a42-880d-af9132134113\") " Sep 15 11:45:02.471689 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:02.471666 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d432c92-1211-4a42-880d-af9132134113-kube-api-access-nffsn" (OuterVolumeSpecName: "kube-api-access-nffsn") pod "2d432c92-1211-4a42-880d-af9132134113" (UID: "2d432c92-1211-4a42-880d-af9132134113"). InnerVolumeSpecName "kube-api-access-nffsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:45:02.571009 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:02.570972 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nffsn\" (UniqueName: \"kubernetes.io/projected/2d432c92-1211-4a42-880d-af9132134113-kube-api-access-nffsn\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:45:03.328152 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:03.328127 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54766dc64c-4pkgn" Sep 15 11:45:03.377252 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:03.377220 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54766dc64c-4pkgn"] Sep 15 11:45:03.382986 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:03.382956 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-54766dc64c-4pkgn"] Sep 15 11:45:04.332988 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:04.332951 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc7f4c957-skzpv" event={"ID":"517f5bf5-39d6-4c7e-a902-717d325bdf77","Type":"ContainerStarted","Data":"830f14cb65ce65c1aacf3b836f74b6c2eb672cfcf975b7fdddbc5dc2dd2e07ad"} Sep 15 11:45:04.349743 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:04.349695 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5fc7f4c957-skzpv" podStartSLOduration=2.309313912 podStartE2EDuration="3.349680247s" totalImagesPullingTime="1.040366335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:04.348888776 +0000 UTC m=+491.578815312" watchObservedRunningTime="2026-09-15 11:45:04.349680247 +0000 UTC m=+491.579606783" Sep 15 11:45:04.373749 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:04.373717 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sdmth"] Sep 15 11:45:04.373966 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:04.373941 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-sdmth" podUID="5b3787cb-fa22-4fdd-84a4-63810e5abd03" containerName="authorino" containerID="cri-o://c375902ad8ee6c7f92996013f72bcf135d43275f9c782a6772f998b466f9e719" gracePeriod=30 Sep 15 11:45:04.614597 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:04.614570 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sdmth" Sep 15 11:45:04.689321 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:04.689272 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfhq7\" (UniqueName: \"kubernetes.io/projected/5b3787cb-fa22-4fdd-84a4-63810e5abd03-kube-api-access-zfhq7\") pod \"5b3787cb-fa22-4fdd-84a4-63810e5abd03\" (UID: \"5b3787cb-fa22-4fdd-84a4-63810e5abd03\") " Sep 15 11:45:04.691301 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:04.691267 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3787cb-fa22-4fdd-84a4-63810e5abd03-kube-api-access-zfhq7" (OuterVolumeSpecName: "kube-api-access-zfhq7") pod "5b3787cb-fa22-4fdd-84a4-63810e5abd03" (UID: "5b3787cb-fa22-4fdd-84a4-63810e5abd03"). InnerVolumeSpecName "kube-api-access-zfhq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:45:04.789946 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:04.789893 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfhq7\" (UniqueName: \"kubernetes.io/projected/5b3787cb-fa22-4fdd-84a4-63810e5abd03-kube-api-access-zfhq7\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:45:05.234689 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:05.234655 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d432c92-1211-4a42-880d-af9132134113" path="/var/lib/kubelet/pods/2d432c92-1211-4a42-880d-af9132134113/volumes" Sep 15 11:45:05.338322 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:05.338285 2573 generic.go:356] "Generic (PLEG): container finished" podID="5b3787cb-fa22-4fdd-84a4-63810e5abd03" containerID="c375902ad8ee6c7f92996013f72bcf135d43275f9c782a6772f998b466f9e719" exitCode=0 Sep 15 11:45:05.338322 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:05.338327 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sdmth" Sep 15 11:45:05.338747 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:05.338372 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sdmth" event={"ID":"5b3787cb-fa22-4fdd-84a4-63810e5abd03","Type":"ContainerDied","Data":"c375902ad8ee6c7f92996013f72bcf135d43275f9c782a6772f998b466f9e719"} Sep 15 11:45:05.338747 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:05.338417 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sdmth" event={"ID":"5b3787cb-fa22-4fdd-84a4-63810e5abd03","Type":"ContainerDied","Data":"b1db63194c3cc5ed62d730d0cdbc87289499f309ec07bf0efcca90ca41845318"} Sep 15 11:45:05.338747 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:05.338437 2573 scope.go:117] "RemoveContainer" containerID="c375902ad8ee6c7f92996013f72bcf135d43275f9c782a6772f998b466f9e719" Sep 15 11:45:05.346292 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:05.346276 2573 scope.go:117] "RemoveContainer" containerID="c375902ad8ee6c7f92996013f72bcf135d43275f9c782a6772f998b466f9e719" Sep 15 11:45:05.346557 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:45:05.346538 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c375902ad8ee6c7f92996013f72bcf135d43275f9c782a6772f998b466f9e719\": container with ID starting with c375902ad8ee6c7f92996013f72bcf135d43275f9c782a6772f998b466f9e719 not found: ID does not exist" containerID="c375902ad8ee6c7f92996013f72bcf135d43275f9c782a6772f998b466f9e719" Sep 15 11:45:05.346610 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:05.346571 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c375902ad8ee6c7f92996013f72bcf135d43275f9c782a6772f998b466f9e719"} err="failed to get container status \"c375902ad8ee6c7f92996013f72bcf135d43275f9c782a6772f998b466f9e719\": rpc error: code = NotFound desc = could not find container \"c375902ad8ee6c7f92996013f72bcf135d43275f9c782a6772f998b466f9e719\": container with ID starting with c375902ad8ee6c7f92996013f72bcf135d43275f9c782a6772f998b466f9e719 not found: ID does not exist" Sep 15 11:45:05.354082 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:05.354056 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sdmth"] Sep 15 11:45:05.358415 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:05.358392 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sdmth"] Sep 15 11:45:07.234286 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:45:07.234244 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b3787cb-fa22-4fdd-84a4-63810e5abd03" path="/var/lib/kubelet/pods/5b3787cb-fa22-4fdd-84a4-63810e5abd03/volumes" Sep 15 11:46:01.698333 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.698287 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd"] Sep 15 11:46:01.699143 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.698752 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b3787cb-fa22-4fdd-84a4-63810e5abd03" containerName="authorino" Sep 15 11:46:01.699143 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.698771 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b3787cb-fa22-4fdd-84a4-63810e5abd03" containerName="authorino" Sep 15 11:46:01.699143 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.698873 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b3787cb-fa22-4fdd-84a4-63810e5abd03" containerName="authorino" Sep 15 11:46:01.703492 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.703469 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.705921 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.705893 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 15 11:46:01.706132 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.706108 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sl78g\"" Sep 15 11:46:01.706876 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.706852 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 15 11:46:01.706973 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.706891 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 15 11:46:01.714144 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.714120 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd"] Sep 15 11:46:01.847281 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.847249 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce79137-d3db-431d-af86-1574b9504736-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.847281 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.847287 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce79137-d3db-431d-af86-1574b9504736-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.847507 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.847316 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce79137-d3db-431d-af86-1574b9504736-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.847507 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.847406 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce79137-d3db-431d-af86-1574b9504736-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.847507 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.847460 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkktz\" (UniqueName: \"kubernetes.io/projected/8ce79137-d3db-431d-af86-1574b9504736-kube-api-access-tkktz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.847507 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.847490 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce79137-d3db-431d-af86-1574b9504736-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.948482 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.948386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce79137-d3db-431d-af86-1574b9504736-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.948482 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.948458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce79137-d3db-431d-af86-1574b9504736-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.948660 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.948535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce79137-d3db-431d-af86-1574b9504736-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.948660 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.948591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkktz\" (UniqueName: \"kubernetes.io/projected/8ce79137-d3db-431d-af86-1574b9504736-kube-api-access-tkktz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.948660 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.948627 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce79137-d3db-431d-af86-1574b9504736-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.948807 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.948685 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce79137-d3db-431d-af86-1574b9504736-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.948807 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.948795 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce79137-d3db-431d-af86-1574b9504736-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.948922 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.948863 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce79137-d3db-431d-af86-1574b9504736-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.949011 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.948992 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce79137-d3db-431d-af86-1574b9504736-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.950931 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.950904 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce79137-d3db-431d-af86-1574b9504736-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.951246 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.951222 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce79137-d3db-431d-af86-1574b9504736-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:01.956924 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:01.956899 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkktz\" (UniqueName: \"kubernetes.io/projected/8ce79137-d3db-431d-af86-1574b9504736-kube-api-access-tkktz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd\" (UID: \"8ce79137-d3db-431d-af86-1574b9504736\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:02.017099 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:02.017066 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:02.288308 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:02.288275 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd"] Sep 15 11:46:02.290398 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:46:02.290370 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce79137_d3db_431d_af86_1574b9504736.slice/crio-0b41b00cc17817f734e10a145c56715cfbc45a7692235f1fb2a7fc249cc9e1c5 WatchSource:0}: Error finding container 0b41b00cc17817f734e10a145c56715cfbc45a7692235f1fb2a7fc249cc9e1c5: Status 404 returned error can't find the container with id 0b41b00cc17817f734e10a145c56715cfbc45a7692235f1fb2a7fc249cc9e1c5 Sep 15 11:46:02.586760 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:02.586671 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" event={"ID":"8ce79137-d3db-431d-af86-1574b9504736","Type":"ContainerStarted","Data":"0b41b00cc17817f734e10a145c56715cfbc45a7692235f1fb2a7fc249cc9e1c5"} Sep 15 11:46:03.592554 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:03.592515 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" event={"ID":"8ce79137-d3db-431d-af86-1574b9504736","Type":"ContainerStarted","Data":"0e68f2a8e52db2fb3031125d516f268a72c63a411f8d9bf157df934ee282861c"} Sep 15 11:46:03.610238 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:03.610188 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" podStartSLOduration=1.536289021 podStartE2EDuration="2.610172028s" totalImagesPullingTime="1.073883007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:46:03.609243909 +0000 UTC m=+550.839170484" watchObservedRunningTime="2026-09-15 11:46:03.610172028 +0000 UTC m=+550.840098567" Sep 15 11:46:12.018035 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:12.017998 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:12.018406 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:12.018087 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:12.030422 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:12.030399 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:12.642499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:12.642464 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-22xzd" Sep 15 11:46:21.465033 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.464993 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9"] Sep 15 11:46:21.468113 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.468094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.470447 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.470426 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 15 11:46:21.478147 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.478126 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9"] Sep 15 11:46:21.610990 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.610952 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.610990 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.610995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnc8l\" (UniqueName: \"kubernetes.io/projected/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-kube-api-access-wnc8l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.611274 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.611052 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.611274 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.611086 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.611274 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.611111 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.611274 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.611173 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.711831 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.711791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.712008 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.711877 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.712008 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.711910 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnc8l\" (UniqueName: \"kubernetes.io/projected/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-kube-api-access-wnc8l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.712008 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.711991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.712215 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.712064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.712215 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.712105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.712401 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.712377 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.712468 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.712440 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.712592 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.712560 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.714141 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.714114 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.714417 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.714399 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.719739 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.719685 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnc8l\" (UniqueName: \"kubernetes.io/projected/65ee0dd1-5f15-426b-8bf3-8fb87908c2ef-kube-api-access-wnc8l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4tk9\" (UID: \"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:21.777748 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:21.777725 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:22.050271 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:22.050145 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9"] Sep 15 11:46:22.052798 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:46:22.052769 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ee0dd1_5f15_426b_8bf3_8fb87908c2ef.slice/crio-9878d2ee422c8da02726e0ec81fcd5066829a932c1b63c7cf0ca8b1d34a61280 WatchSource:0}: Error finding container 9878d2ee422c8da02726e0ec81fcd5066829a932c1b63c7cf0ca8b1d34a61280: Status 404 returned error can't find the container with id 9878d2ee422c8da02726e0ec81fcd5066829a932c1b63c7cf0ca8b1d34a61280 Sep 15 11:46:22.677018 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:22.676983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" event={"ID":"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef","Type":"ContainerStarted","Data":"a3b424b5d4671b63cdea3e756843822ecd34227db4398e5e360652c9531c6f40"} Sep 15 11:46:22.677395 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:22.677056 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" event={"ID":"65ee0dd1-5f15-426b-8bf3-8fb87908c2ef","Type":"ContainerStarted","Data":"9878d2ee422c8da02726e0ec81fcd5066829a932c1b63c7cf0ca8b1d34a61280"} Sep 15 11:46:22.695765 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:22.695688 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" podStartSLOduration=1.502543843 podStartE2EDuration="1.695666132s" totalImagesPullingTime="193.122289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:46:22.694831342 +0000 UTC m=+569.924757875" watchObservedRunningTime="2026-09-15 11:46:22.695666132 +0000 UTC m=+569.925592668" Sep 15 11:46:31.778267 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:31.778228 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:31.778587 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:31.778292 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:31.790893 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:31.790873 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:32.731208 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:32.731177 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4tk9" Sep 15 11:46:35.575823 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.575790 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz"] Sep 15 11:46:35.579191 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.579174 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.581434 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.581417 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 15 11:46:35.589053 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.589019 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz"] Sep 15 11:46:35.617128 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.617102 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.617217 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.617133 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.617217 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.617148 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.617217 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.617188 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.617320 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.617244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pddpp\" (UniqueName: \"kubernetes.io/projected/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-kube-api-access-pddpp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.617320 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.617268 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.718344 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.718314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pddpp\" (UniqueName: \"kubernetes.io/projected/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-kube-api-access-pddpp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.718465 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.718354 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.718465 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.718402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.718465 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.718422 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.718465 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.718437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.718465 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.718458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.718786 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.718762 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.718868 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.718826 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.718915 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.718894 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.720654 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.720633 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.720976 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.720959 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.725627 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.725604 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pddpp\" (UniqueName: \"kubernetes.io/projected/7f4f13f4-eee1-4762-9daa-63fe6e0edb77-kube-api-access-pddpp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz\" (UID: \"7f4f13f4-eee1-4762-9daa-63fe6e0edb77\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:35.889733 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:35.889656 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:36.159161 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:36.159136 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz"] Sep 15 11:46:36.161168 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:46:36.161132 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4f13f4_eee1_4762_9daa_63fe6e0edb77.slice/crio-e16b0031e442a958f2218b71721083533843d5449542f50f421f4cc3e4658393 WatchSource:0}: Error finding container e16b0031e442a958f2218b71721083533843d5449542f50f421f4cc3e4658393: Status 404 returned error can't find the container with id e16b0031e442a958f2218b71721083533843d5449542f50f421f4cc3e4658393 Sep 15 11:46:36.739200 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:36.739118 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" event={"ID":"7f4f13f4-eee1-4762-9daa-63fe6e0edb77","Type":"ContainerStarted","Data":"e6c7595710198e204ea575b3404239b6ff64d2262fb6b9230a015567520b9b47"} Sep 15 11:46:36.739200 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:36.739167 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" event={"ID":"7f4f13f4-eee1-4762-9daa-63fe6e0edb77","Type":"ContainerStarted","Data":"e16b0031e442a958f2218b71721083533843d5449542f50f421f4cc3e4658393"} Sep 15 11:46:36.756952 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:36.756900 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" podStartSLOduration=1.552488729 podStartE2EDuration="1.756883547s" totalImagesPullingTime="204.394818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:46:36.755346122 +0000 UTC m=+583.985272661" watchObservedRunningTime="2026-09-15 11:46:36.756883547 +0000 UTC m=+583.986810133" Sep 15 11:46:43.465148 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.465110 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5"] Sep 15 11:46:43.469043 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.469003 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.471302 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.471282 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 15 11:46:43.479291 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.479195 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5"] Sep 15 11:46:43.480064 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.480042 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.480176 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.480092 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.480176 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.480118 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.480255 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.480195 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.480295 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.480260 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdjdj\" (UniqueName: \"kubernetes.io/projected/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-kube-api-access-zdjdj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.480343 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.480291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.581407 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.581373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.581407 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.581408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.581638 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.581431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.581638 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.581455 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.581638 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.581479 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdjdj\" (UniqueName: \"kubernetes.io/projected/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-kube-api-access-zdjdj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.581638 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.581495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.581872 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.581842 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.581932 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.581905 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.581991 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.581970 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.583673 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.583644 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.583976 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.583956 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.588901 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.588884 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdjdj\" (UniqueName: \"kubernetes.io/projected/c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90-kube-api-access-zdjdj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dx6n5\" (UID: \"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:43.783970 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:43.783939 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:44.052882 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:44.052858 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5"] Sep 15 11:46:44.055762 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:46:44.055732 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ac4a18_f95b_43f9_ac04_f6c4b6a7da90.slice/crio-9d1e410286b6979e79797fd39c818d6ffc225b5b6af7dd6ce0e971ddd6a2dc8e WatchSource:0}: Error finding container 9d1e410286b6979e79797fd39c818d6ffc225b5b6af7dd6ce0e971ddd6a2dc8e: Status 404 returned error can't find the container with id 9d1e410286b6979e79797fd39c818d6ffc225b5b6af7dd6ce0e971ddd6a2dc8e Sep 15 11:46:44.775272 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:44.775225 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" event={"ID":"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90","Type":"ContainerStarted","Data":"2353f24acd4c4ed35ac7cdc456777ada8d71bb6da8f22d53661c28822c32c313"} Sep 15 11:46:44.775663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:44.775328 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" event={"ID":"c0ac4a18-f95b-43f9-ac04-f6c4b6a7da90","Type":"ContainerStarted","Data":"9d1e410286b6979e79797fd39c818d6ffc225b5b6af7dd6ce0e971ddd6a2dc8e"} Sep 15 11:46:44.792318 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:44.792268 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" podStartSLOduration=1.579594704 podStartE2EDuration="1.792253252s" totalImagesPullingTime="212.658548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:46:44.790923409 +0000 UTC m=+592.020849963" watchObservedRunningTime="2026-09-15 11:46:44.792253252 +0000 UTC m=+592.022179787" Sep 15 11:46:45.890349 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:45.890313 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:45.890796 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:45.890378 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:45.907304 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:45.907265 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:46.795862 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:46.795832 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjnljz" Sep 15 11:46:53.207305 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:53.207277 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 11:46:53.207770 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:53.207394 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 11:46:53.218017 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:53.217996 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 11:46:53.218170 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:53.218057 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 11:46:53.784604 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:53.784563 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:53.786771 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:53.784640 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:53.797451 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:53.797427 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:46:53.832441 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:46:53.832409 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dx6n5" Sep 15 11:48:29.788681 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:29.788643 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fc7f4c957-skzpv"] Sep 15 11:48:29.789192 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:29.788818 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5fc7f4c957-skzpv" podUID="517f5bf5-39d6-4c7e-a902-717d325bdf77" containerName="authorino" containerID="cri-o://830f14cb65ce65c1aacf3b836f74b6c2eb672cfcf975b7fdddbc5dc2dd2e07ad" gracePeriod=30 Sep 15 11:48:30.026821 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.026797 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc7f4c957-skzpv" Sep 15 11:48:30.081656 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.081578 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh2sj\" (UniqueName: \"kubernetes.io/projected/517f5bf5-39d6-4c7e-a902-717d325bdf77-kube-api-access-wh2sj\") pod \"517f5bf5-39d6-4c7e-a902-717d325bdf77\" (UID: \"517f5bf5-39d6-4c7e-a902-717d325bdf77\") " Sep 15 11:48:30.081656 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.081629 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/517f5bf5-39d6-4c7e-a902-717d325bdf77-tls-cert\") pod \"517f5bf5-39d6-4c7e-a902-717d325bdf77\" (UID: \"517f5bf5-39d6-4c7e-a902-717d325bdf77\") " Sep 15 11:48:30.083667 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.083638 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517f5bf5-39d6-4c7e-a902-717d325bdf77-kube-api-access-wh2sj" (OuterVolumeSpecName: "kube-api-access-wh2sj") pod "517f5bf5-39d6-4c7e-a902-717d325bdf77" (UID: "517f5bf5-39d6-4c7e-a902-717d325bdf77"). InnerVolumeSpecName "kube-api-access-wh2sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:48:30.091349 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.091322 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/517f5bf5-39d6-4c7e-a902-717d325bdf77-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "517f5bf5-39d6-4c7e-a902-717d325bdf77" (UID: "517f5bf5-39d6-4c7e-a902-717d325bdf77"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:48:30.182558 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.182526 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh2sj\" (UniqueName: \"kubernetes.io/projected/517f5bf5-39d6-4c7e-a902-717d325bdf77-kube-api-access-wh2sj\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:48:30.182558 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.182554 2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/517f5bf5-39d6-4c7e-a902-717d325bdf77-tls-cert\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:48:30.243345 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.243313 2573 generic.go:356] "Generic (PLEG): container finished" podID="517f5bf5-39d6-4c7e-a902-717d325bdf77" containerID="830f14cb65ce65c1aacf3b836f74b6c2eb672cfcf975b7fdddbc5dc2dd2e07ad" exitCode=0 Sep 15 11:48:30.243478 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.243352 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc7f4c957-skzpv" Sep 15 11:48:30.243478 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.243379 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc7f4c957-skzpv" event={"ID":"517f5bf5-39d6-4c7e-a902-717d325bdf77","Type":"ContainerDied","Data":"830f14cb65ce65c1aacf3b836f74b6c2eb672cfcf975b7fdddbc5dc2dd2e07ad"} Sep 15 11:48:30.243478 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.243408 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc7f4c957-skzpv" event={"ID":"517f5bf5-39d6-4c7e-a902-717d325bdf77","Type":"ContainerDied","Data":"a039e98fc1c2b0b785dba9c68851b91fe6f3991a4b2f4af286989d3d3883304b"} Sep 15 11:48:30.243478 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.243426 2573 scope.go:117] "RemoveContainer" containerID="830f14cb65ce65c1aacf3b836f74b6c2eb672cfcf975b7fdddbc5dc2dd2e07ad" Sep 15 11:48:30.252176 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.252160 2573 scope.go:117] "RemoveContainer" containerID="830f14cb65ce65c1aacf3b836f74b6c2eb672cfcf975b7fdddbc5dc2dd2e07ad" Sep 15 11:48:30.252467 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:48:30.252449 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830f14cb65ce65c1aacf3b836f74b6c2eb672cfcf975b7fdddbc5dc2dd2e07ad\": container with ID starting with 830f14cb65ce65c1aacf3b836f74b6c2eb672cfcf975b7fdddbc5dc2dd2e07ad not found: ID does not exist" containerID="830f14cb65ce65c1aacf3b836f74b6c2eb672cfcf975b7fdddbc5dc2dd2e07ad" Sep 15 11:48:30.252515 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.252478 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830f14cb65ce65c1aacf3b836f74b6c2eb672cfcf975b7fdddbc5dc2dd2e07ad"} err="failed to get container status \"830f14cb65ce65c1aacf3b836f74b6c2eb672cfcf975b7fdddbc5dc2dd2e07ad\": rpc error: code = NotFound desc = could not find container \"830f14cb65ce65c1aacf3b836f74b6c2eb672cfcf975b7fdddbc5dc2dd2e07ad\": container with ID starting with 830f14cb65ce65c1aacf3b836f74b6c2eb672cfcf975b7fdddbc5dc2dd2e07ad not found: ID does not exist" Sep 15 11:48:30.264599 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.264579 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fc7f4c957-skzpv"] Sep 15 11:48:30.268309 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:30.268289 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5fc7f4c957-skzpv"] Sep 15 11:48:31.234680 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:48:31.234639 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="517f5bf5-39d6-4c7e-a902-717d325bdf77" path="/var/lib/kubelet/pods/517f5bf5-39d6-4c7e-a902-717d325bdf77/volumes" Sep 15 11:50:09.523539 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:09.523500 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7104-25d81e"] Sep 15 11:50:09.523973 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:09.523821 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517f5bf5-39d6-4c7e-a902-717d325bdf77" containerName="authorino" Sep 15 11:50:09.523973 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:09.523834 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="517f5bf5-39d6-4c7e-a902-717d325bdf77" containerName="authorino" Sep 15 11:50:09.523973 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:09.523913 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="517f5bf5-39d6-4c7e-a902-717d325bdf77" containerName="authorino" Sep 15 11:50:09.526843 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:09.526825 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-25d81e" Sep 15 11:50:09.529405 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:09.529383 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dg4tz\"" Sep 15 11:50:09.545034 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:09.544989 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-25d81e"] Sep 15 11:50:09.628069 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:09.628008 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdhd\" (UniqueName: \"kubernetes.io/projected/bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20-kube-api-access-jzdhd\") pod \"test-curl-7104-25d81e\" (UID: \"bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20\") " pod="opendatahub/test-curl-7104-25d81e" Sep 15 11:50:09.728860 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:09.728817 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdhd\" (UniqueName: \"kubernetes.io/projected/bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20-kube-api-access-jzdhd\") pod \"test-curl-7104-25d81e\" (UID: \"bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20\") " pod="opendatahub/test-curl-7104-25d81e" Sep 15 11:50:09.751490 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:09.751459 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdhd\" (UniqueName: \"kubernetes.io/projected/bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20-kube-api-access-jzdhd\") pod \"test-curl-7104-25d81e\" (UID: \"bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20\") " pod="opendatahub/test-curl-7104-25d81e" Sep 15 11:50:09.835585 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:09.835502 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-25d81e" Sep 15 11:50:10.100948 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:10.100865 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-25d81e"] Sep 15 11:50:10.103304 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:50:10.103274 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd3c0c5e_be7a_4c64_a0b2_af2141ae0b20.slice/crio-2de7a0f082170b0697ef7928ddaa417730c614a38d2105469057583ca5eb15a5 WatchSource:0}: Error finding container 2de7a0f082170b0697ef7928ddaa417730c614a38d2105469057583ca5eb15a5: Status 404 returned error can't find the container with id 2de7a0f082170b0697ef7928ddaa417730c614a38d2105469057583ca5eb15a5 Sep 15 11:50:10.105175 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:10.105157 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 11:50:10.672413 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:10.672378 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-25d81e" event={"ID":"bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20","Type":"ContainerStarted","Data":"2de7a0f082170b0697ef7928ddaa417730c614a38d2105469057583ca5eb15a5"} Sep 15 11:50:12.684351 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:12.684318 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-25d81e" event={"ID":"bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20","Type":"ContainerStarted","Data":"07186a369b00cd02f3b40bf1ff68b5dddd52d8c64ec456a3e0590b876d7606e5"} Sep 15 11:50:12.705964 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:12.705916 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7104-25d81e" podStartSLOduration=1.7363867819999999 podStartE2EDuration="3.705900593s" totalImagesPullingTime="1.969513811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:50:12.704841746 +0000 UTC m=+799.934768282" watchObservedRunningTime="2026-09-15 11:50:12.705900593 +0000 UTC m=+799.935827129" Sep 15 11:50:13.412333 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:13.412301 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7104-25d81e"] Sep 15 11:50:13.416330 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:13.416258 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7104-25d81e"] Sep 15 11:50:13.565150 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:13.565108 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7104-318dc7"] Sep 15 11:50:13.565503 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:13.565484 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20" containerName="test-curl-7104-25d81e" Sep 15 11:50:13.565591 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:13.565506 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20" containerName="test-curl-7104-25d81e" Sep 15 11:50:13.565644 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:13.565615 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20" containerName="test-curl-7104-25d81e" Sep 15 11:50:13.568505 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:13.568483 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-318dc7" Sep 15 11:50:13.573010 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:13.572986 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-318dc7"] Sep 15 11:50:13.660574 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:13.660534 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fv8d\" (UniqueName: \"kubernetes.io/projected/5edf71f4-57bf-4ce4-adc5-771ff9205101-kube-api-access-2fv8d\") pod \"test-curl-7104-318dc7\" (UID: \"5edf71f4-57bf-4ce4-adc5-771ff9205101\") " pod="opendatahub/test-curl-7104-318dc7" Sep 15 11:50:13.690335 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:13.690259 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7104-25d81e" container="test-curl-7104-25d81e" Sep 15 11:50:13.761704 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:13.761670 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fv8d\" (UniqueName: \"kubernetes.io/projected/5edf71f4-57bf-4ce4-adc5-771ff9205101-kube-api-access-2fv8d\") pod \"test-curl-7104-318dc7\" (UID: \"5edf71f4-57bf-4ce4-adc5-771ff9205101\") " pod="opendatahub/test-curl-7104-318dc7" Sep 15 11:50:13.769289 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:13.769267 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fv8d\" (UniqueName: \"kubernetes.io/projected/5edf71f4-57bf-4ce4-adc5-771ff9205101-kube-api-access-2fv8d\") pod \"test-curl-7104-318dc7\" (UID: \"5edf71f4-57bf-4ce4-adc5-771ff9205101\") " pod="opendatahub/test-curl-7104-318dc7" Sep 15 11:50:13.878674 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:13.878622 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-318dc7" Sep 15 11:50:14.145912 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:14.145882 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-318dc7"] Sep 15 11:50:14.695722 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:14.695676 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-318dc7" event={"ID":"5edf71f4-57bf-4ce4-adc5-771ff9205101","Type":"ContainerStarted","Data":"71624713c5c047f903027e007864f07f5b4ac8c8f1a8e4bc17480bb2d1f1f7f5"} Sep 15 11:50:14.696215 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:14.695731 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-318dc7" event={"ID":"5edf71f4-57bf-4ce4-adc5-771ff9205101","Type":"ContainerStarted","Data":"e8601e23315870b092d7ad81e35256c9c8812f0d00e9febcea5d8d4c19030bcf"} Sep 15 11:50:14.696215 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:14.695756 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7104-25d81e" podUID="bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20" containerName="test-curl-7104-25d81e" containerID="cri-o://07186a369b00cd02f3b40bf1ff68b5dddd52d8c64ec456a3e0590b876d7606e5" gracePeriod=2 Sep 15 11:50:14.710046 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:14.709981 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7104-318dc7" podStartSLOduration=1.7099650579999999 podStartE2EDuration="1.709965058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:50:14.709296443 +0000 UTC m=+801.939222980" watchObservedRunningTime="2026-09-15 11:50:14.709965058 +0000 UTC m=+801.939891593" Sep 15 11:50:14.711186 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:14.711154 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-25d81e\" is forbidden: User \"system:node:ip-10-0-128-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-189.ec2.internal' and this object" podUID="bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20" pod="opendatahub/test-curl-7104-25d81e" Sep 15 11:50:14.712936 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:14.712912 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-25d81e\" is forbidden: User \"system:node:ip-10-0-128-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-189.ec2.internal' and this object" podUID="bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20" pod="opendatahub/test-curl-7104-25d81e" Sep 15 11:50:15.364373 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:15.364338 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7104-318dc7"] Sep 15 11:50:15.368470 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:15.368441 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7104-318dc7"] Sep 15 11:50:15.699744 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:15.699712 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7104-318dc7" container="test-curl-7104-318dc7" Sep 15 11:50:16.704385 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:16.704357 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7104-318dc7" podUID="5edf71f4-57bf-4ce4-adc5-771ff9205101" containerName="test-curl-7104-318dc7" containerID="cri-o://71624713c5c047f903027e007864f07f5b4ac8c8f1a8e4bc17480bb2d1f1f7f5" gracePeriod=2 Sep 15 11:50:16.706676 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:16.706639 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-318dc7\" is forbidden: User \"system:node:ip-10-0-128-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-189.ec2.internal' and this object" podUID="5edf71f4-57bf-4ce4-adc5-771ff9205101" pod="opendatahub/test-curl-7104-318dc7" Sep 15 11:50:16.869488 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:16.869462 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-25d81e" Sep 15 11:50:16.871256 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:16.871232 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-318dc7\" is forbidden: User \"system:node:ip-10-0-128-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-189.ec2.internal' and this object" podUID="5edf71f4-57bf-4ce4-adc5-771ff9205101" pod="opendatahub/test-curl-7104-318dc7" Sep 15 11:50:16.881521 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:16.881502 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzdhd\" (UniqueName: \"kubernetes.io/projected/bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20-kube-api-access-jzdhd\") pod \"bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20\" (UID: \"bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20\") " Sep 15 11:50:16.883515 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:16.883493 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20-kube-api-access-jzdhd" (OuterVolumeSpecName: "kube-api-access-jzdhd") pod "bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20" (UID: "bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20"). InnerVolumeSpecName "kube-api-access-jzdhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:50:16.982144 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:16.982070 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzdhd\" (UniqueName: \"kubernetes.io/projected/bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20-kube-api-access-jzdhd\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:50:17.234579 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:17.234497 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20" path="/var/lib/kubelet/pods/bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20/volumes" Sep 15 11:50:17.709974 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:17.709933 2573 generic.go:356] "Generic (PLEG): container finished" podID="bd3c0c5e-be7a-4c64-a0b2-af2141ae0b20" containerID="07186a369b00cd02f3b40bf1ff68b5dddd52d8c64ec456a3e0590b876d7606e5" exitCode=137 Sep 15 11:50:17.710445 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:17.709981 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-25d81e" Sep 15 11:50:17.710445 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:17.710053 2573 scope.go:117] "RemoveContainer" containerID="07186a369b00cd02f3b40bf1ff68b5dddd52d8c64ec456a3e0590b876d7606e5" Sep 15 11:50:17.718169 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:17.718148 2573 scope.go:117] "RemoveContainer" containerID="07186a369b00cd02f3b40bf1ff68b5dddd52d8c64ec456a3e0590b876d7606e5" Sep 15 11:50:17.718476 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:17.718454 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07186a369b00cd02f3b40bf1ff68b5dddd52d8c64ec456a3e0590b876d7606e5\": container with ID starting with 07186a369b00cd02f3b40bf1ff68b5dddd52d8c64ec456a3e0590b876d7606e5 not found: ID does not exist" containerID="07186a369b00cd02f3b40bf1ff68b5dddd52d8c64ec456a3e0590b876d7606e5" Sep 15 11:50:17.718537 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:17.718485 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07186a369b00cd02f3b40bf1ff68b5dddd52d8c64ec456a3e0590b876d7606e5"} err="failed to get container status \"07186a369b00cd02f3b40bf1ff68b5dddd52d8c64ec456a3e0590b876d7606e5\": rpc error: code = NotFound desc = could not find container \"07186a369b00cd02f3b40bf1ff68b5dddd52d8c64ec456a3e0590b876d7606e5\": container with ID starting with 07186a369b00cd02f3b40bf1ff68b5dddd52d8c64ec456a3e0590b876d7606e5 not found: ID does not exist" Sep 15 11:50:18.875823 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:18.875798 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-318dc7" Sep 15 11:50:18.896739 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:18.896715 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fv8d\" (UniqueName: \"kubernetes.io/projected/5edf71f4-57bf-4ce4-adc5-771ff9205101-kube-api-access-2fv8d\") pod \"5edf71f4-57bf-4ce4-adc5-771ff9205101\" (UID: \"5edf71f4-57bf-4ce4-adc5-771ff9205101\") " Sep 15 11:50:18.898671 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:18.898642 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edf71f4-57bf-4ce4-adc5-771ff9205101-kube-api-access-2fv8d" (OuterVolumeSpecName: "kube-api-access-2fv8d") pod "5edf71f4-57bf-4ce4-adc5-771ff9205101" (UID: "5edf71f4-57bf-4ce4-adc5-771ff9205101"). InnerVolumeSpecName "kube-api-access-2fv8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:50:18.998132 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:18.998056 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fv8d\" (UniqueName: \"kubernetes.io/projected/5edf71f4-57bf-4ce4-adc5-771ff9205101-kube-api-access-2fv8d\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:50:19.235134 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:19.235099 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5edf71f4-57bf-4ce4-adc5-771ff9205101" path="/var/lib/kubelet/pods/5edf71f4-57bf-4ce4-adc5-771ff9205101/volumes" Sep 15 11:50:19.721392 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:19.721354 2573 generic.go:356] "Generic (PLEG): container finished" podID="5edf71f4-57bf-4ce4-adc5-771ff9205101" containerID="71624713c5c047f903027e007864f07f5b4ac8c8f1a8e4bc17480bb2d1f1f7f5" exitCode=137 Sep 15 11:50:19.721548 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:19.721397 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-318dc7" Sep 15 11:50:19.721548 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:19.721441 2573 scope.go:117] "RemoveContainer" containerID="71624713c5c047f903027e007864f07f5b4ac8c8f1a8e4bc17480bb2d1f1f7f5" Sep 15 11:50:19.729919 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:19.729897 2573 scope.go:117] "RemoveContainer" containerID="71624713c5c047f903027e007864f07f5b4ac8c8f1a8e4bc17480bb2d1f1f7f5" Sep 15 11:50:19.730213 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:19.730190 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71624713c5c047f903027e007864f07f5b4ac8c8f1a8e4bc17480bb2d1f1f7f5\": container with ID starting with 71624713c5c047f903027e007864f07f5b4ac8c8f1a8e4bc17480bb2d1f1f7f5 not found: ID does not exist" containerID="71624713c5c047f903027e007864f07f5b4ac8c8f1a8e4bc17480bb2d1f1f7f5" Sep 15 11:50:19.730257 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:19.730221 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71624713c5c047f903027e007864f07f5b4ac8c8f1a8e4bc17480bb2d1f1f7f5"} err="failed to get container status \"71624713c5c047f903027e007864f07f5b4ac8c8f1a8e4bc17480bb2d1f1f7f5\": rpc error: code = NotFound desc = could not find container \"71624713c5c047f903027e007864f07f5b4ac8c8f1a8e4bc17480bb2d1f1f7f5\": container with ID starting with 71624713c5c047f903027e007864f07f5b4ac8c8f1a8e4bc17480bb2d1f1f7f5 not found: ID does not exist" Sep 15 11:50:20.856683 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.856646 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65"] Sep 15 11:50:20.857045 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.857033 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5edf71f4-57bf-4ce4-adc5-771ff9205101" containerName="test-curl-7104-318dc7" Sep 15 11:50:20.857103 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.857046 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5edf71f4-57bf-4ce4-adc5-771ff9205101" containerName="test-curl-7104-318dc7" Sep 15 11:50:20.857139 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.857110 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5edf71f4-57bf-4ce4-adc5-771ff9205101" containerName="test-curl-7104-318dc7" Sep 15 11:50:20.861121 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.861096 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" Sep 15 11:50:20.863527 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.863509 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ee96ca62\"" Sep 15 11:50:20.863626 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.863510 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ee96ca62-dockercfg-lbf7l\"" Sep 15 11:50:20.868739 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.868718 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65"] Sep 15 11:50:20.884524 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.884494 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt"] Sep 15 11:50:20.887567 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.887550 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" Sep 15 11:50:20.895265 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.895242 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt"] Sep 15 11:50:20.914148 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.914115 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98727\" (UniqueName: \"kubernetes.io/projected/f13c65d5-c679-49ea-9a9a-0681ab176dea-kube-api-access-98727\") pod \"payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt\" (UID: \"f13c65d5-c679-49ea-9a9a-0681ab176dea\") " pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" Sep 15 11:50:20.914277 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.914153 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krxlm\" (UniqueName: \"kubernetes.io/projected/36155bfc-7e1a-495e-a388-ecb25881d37e-kube-api-access-krxlm\") pod \"payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65\" (UID: \"36155bfc-7e1a-495e-a388-ecb25881d37e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" Sep 15 11:50:20.914277 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.914186 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13c65d5-c679-49ea-9a9a-0681ab176dea-plugins-config-volume\") pod \"payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt\" (UID: \"f13c65d5-c679-49ea-9a9a-0681ab176dea\") " pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" Sep 15 11:50:20.914366 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:20.914312 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36155bfc-7e1a-495e-a388-ecb25881d37e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65\" (UID: \"36155bfc-7e1a-495e-a388-ecb25881d37e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" Sep 15 11:50:21.015759 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.015713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98727\" (UniqueName: \"kubernetes.io/projected/f13c65d5-c679-49ea-9a9a-0681ab176dea-kube-api-access-98727\") pod \"payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt\" (UID: \"f13c65d5-c679-49ea-9a9a-0681ab176dea\") " pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" Sep 15 11:50:21.015925 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.015765 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krxlm\" (UniqueName: \"kubernetes.io/projected/36155bfc-7e1a-495e-a388-ecb25881d37e-kube-api-access-krxlm\") pod \"payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65\" (UID: \"36155bfc-7e1a-495e-a388-ecb25881d37e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" Sep 15 11:50:21.015925 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.015798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13c65d5-c679-49ea-9a9a-0681ab176dea-plugins-config-volume\") pod \"payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt\" (UID: \"f13c65d5-c679-49ea-9a9a-0681ab176dea\") " pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" Sep 15 11:50:21.015925 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.015833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36155bfc-7e1a-495e-a388-ecb25881d37e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65\" (UID: \"36155bfc-7e1a-495e-a388-ecb25881d37e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" Sep 15 11:50:21.016472 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.016455 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36155bfc-7e1a-495e-a388-ecb25881d37e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65\" (UID: \"36155bfc-7e1a-495e-a388-ecb25881d37e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" Sep 15 11:50:21.016569 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.016550 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13c65d5-c679-49ea-9a9a-0681ab176dea-plugins-config-volume\") pod \"payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt\" (UID: \"f13c65d5-c679-49ea-9a9a-0681ab176dea\") " pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" Sep 15 11:50:21.023751 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.023723 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krxlm\" (UniqueName: \"kubernetes.io/projected/36155bfc-7e1a-495e-a388-ecb25881d37e-kube-api-access-krxlm\") pod \"payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65\" (UID: \"36155bfc-7e1a-495e-a388-ecb25881d37e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" Sep 15 11:50:21.023751 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.023723 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98727\" (UniqueName: \"kubernetes.io/projected/f13c65d5-c679-49ea-9a9a-0681ab176dea-kube-api-access-98727\") pod \"payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt\" (UID: \"f13c65d5-c679-49ea-9a9a-0681ab176dea\") " pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" Sep 15 11:50:21.172053 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.171955 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" Sep 15 11:50:21.198261 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.198227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" Sep 15 11:50:21.444447 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.444285 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65"] Sep 15 11:50:21.446559 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:50:21.446537 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36155bfc_7e1a_495e_a388_ecb25881d37e.slice/crio-4e1ca4214a9d4a0e11e090f8a25b8335fbbc34c149c3aec4d58d46a4285ed5d5 WatchSource:0}: Error finding container 4e1ca4214a9d4a0e11e090f8a25b8335fbbc34c149c3aec4d58d46a4285ed5d5: Status 404 returned error can't find the container with id 4e1ca4214a9d4a0e11e090f8a25b8335fbbc34c149c3aec4d58d46a4285ed5d5 Sep 15 11:50:21.480938 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.480912 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt"] Sep 15 11:50:21.483260 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:50:21.483238 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13c65d5_c679_49ea_9a9a_0681ab176dea.slice/crio-82cbfc722cdc1cebf012b37a85ae05f4b667c837e1035e0464ac0d838139c99f WatchSource:0}: Error finding container 82cbfc722cdc1cebf012b37a85ae05f4b667c837e1035e0464ac0d838139c99f: Status 404 returned error can't find the container with id 82cbfc722cdc1cebf012b37a85ae05f4b667c837e1035e0464ac0d838139c99f Sep 15 11:50:21.732985 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.732897 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" event={"ID":"f13c65d5-c679-49ea-9a9a-0681ab176dea","Type":"ContainerStarted","Data":"82cbfc722cdc1cebf012b37a85ae05f4b667c837e1035e0464ac0d838139c99f"} Sep 15 11:50:21.734052 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:21.734013 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" event={"ID":"36155bfc-7e1a-495e-a388-ecb25881d37e","Type":"ContainerStarted","Data":"4e1ca4214a9d4a0e11e090f8a25b8335fbbc34c149c3aec4d58d46a4285ed5d5"} Sep 15 11:50:24.750674 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:24.750605 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" event={"ID":"f13c65d5-c679-49ea-9a9a-0681ab176dea","Type":"ContainerStarted","Data":"2b0e5b52b997d8bbb924f23df454fa1ec7c997003e1ccb19ab278da7f9747c23"} Sep 15 11:50:24.751210 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:24.750906 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" Sep 15 11:50:24.752495 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:24.752472 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" event={"ID":"36155bfc-7e1a-495e-a388-ecb25881d37e","Type":"ContainerStarted","Data":"83ade6e1f1e4eebde63752b5a9725d0424ac34cd99b483e62eb4d7bea266e59b"} Sep 15 11:50:24.752600 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:24.752516 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" Sep 15 11:50:24.769357 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:24.769314 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" podStartSLOduration=2.492953786 podStartE2EDuration="4.769298226s" totalImagesPullingTime="2.27634444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:50:24.767110125 +0000 UTC m=+811.997036661" watchObservedRunningTime="2026-09-15 11:50:24.769298226 +0000 UTC m=+811.999224759" Sep 15 11:50:24.786462 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:24.786406 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" podStartSLOduration=2.480328331 podStartE2EDuration="4.786391713s" totalImagesPullingTime="2.306063382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:50:24.784304504 +0000 UTC m=+812.014231040" watchObservedRunningTime="2026-09-15 11:50:24.786391713 +0000 UTC m=+812.016318275" Sep 15 11:50:35.759338 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:35.759306 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" Sep 15 11:50:35.759752 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:35.759378 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" Sep 15 11:50:41.554375 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.554344 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn"] Sep 15 11:50:41.558842 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.558821 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:41.561966 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.561941 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1b147a-serving-cert\"" Sep 15 11:50:41.562102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.562009 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 15 11:50:41.562102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.562013 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9dhbv\"" Sep 15 11:50:41.562102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.562060 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 11:50:41.562102 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.562091 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 15 11:50:41.568778 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.568721 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn"] Sep 15 11:50:41.686704 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.686659 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc8a54d-60ec-4d88-aa20-1da7b375d912-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:41.686863 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.686724 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc8a54d-60ec-4d88-aa20-1da7b375d912-metrics-certs\") pod \"maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:41.686863 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.686743 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-262xj\" (UniqueName: \"kubernetes.io/projected/6cc8a54d-60ec-4d88-aa20-1da7b375d912-kube-api-access-262xj\") pod \"maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:41.686863 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.686836 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc8a54d-60ec-4d88-aa20-1da7b375d912-maas-api-tls\") pod \"maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:41.788286 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.788248 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc8a54d-60ec-4d88-aa20-1da7b375d912-metrics-certs\") pod \"maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:41.788438 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.788299 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-262xj\" (UniqueName: \"kubernetes.io/projected/6cc8a54d-60ec-4d88-aa20-1da7b375d912-kube-api-access-262xj\") pod \"maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:41.788438 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.788370 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc8a54d-60ec-4d88-aa20-1da7b375d912-maas-api-tls\") pod \"maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:41.788438 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.788419 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc8a54d-60ec-4d88-aa20-1da7b375d912-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:41.789135 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.789111 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc8a54d-60ec-4d88-aa20-1da7b375d912-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:41.790842 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.790818 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc8a54d-60ec-4d88-aa20-1da7b375d912-metrics-certs\") pod \"maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:41.790936 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.790872 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc8a54d-60ec-4d88-aa20-1da7b375d912-maas-api-tls\") pod \"maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:41.797059 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.797015 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-262xj\" (UniqueName: \"kubernetes.io/projected/6cc8a54d-60ec-4d88-aa20-1da7b375d912-kube-api-access-262xj\") pod \"maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:41.870970 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:41.870907 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:42.148284 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:42.148253 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn"] Sep 15 11:50:42.150375 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:50:42.150350 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc8a54d_60ec_4d88_aa20_1da7b375d912.slice/crio-baecaeb4e01464ab701a00c239c3eb456eca912a0dd32bc39f24634d1b8c3307 WatchSource:0}: Error finding container baecaeb4e01464ab701a00c239c3eb456eca912a0dd32bc39f24634d1b8c3307: Status 404 returned error can't find the container with id baecaeb4e01464ab701a00c239c3eb456eca912a0dd32bc39f24634d1b8c3307 Sep 15 11:50:42.839912 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:42.839870 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" event={"ID":"6cc8a54d-60ec-4d88-aa20-1da7b375d912","Type":"ContainerStarted","Data":"baecaeb4e01464ab701a00c239c3eb456eca912a0dd32bc39f24634d1b8c3307"} Sep 15 11:50:44.851210 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:44.851176 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" event={"ID":"6cc8a54d-60ec-4d88-aa20-1da7b375d912","Type":"ContainerStarted","Data":"96d990507f050f2e65c68402ca808ec7040e24fc9c60474551959a062a63b274"} Sep 15 11:50:44.851660 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:44.851243 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:44.867839 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:44.867785 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" podStartSLOduration=1.546352607 podStartE2EDuration="3.867766063s" totalImagesPullingTime="2.321413456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:50:44.866919632 +0000 UTC m=+832.096846169" watchObservedRunningTime="2026-09-15 11:50:44.867766063 +0000 UTC m=+832.097692601" Sep 15 11:50:50.861248 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:50.861216 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:50:50.875909 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:50.875874 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt"] Sep 15 11:50:50.876144 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:50.876115 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" podUID="f13c65d5-c679-49ea-9a9a-0681ab176dea" containerName="payload-processing" containerID="cri-o://2b0e5b52b997d8bbb924f23df454fa1ec7c997003e1ccb19ab278da7f9747c23" gracePeriod=30 Sep 15 11:50:50.882465 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:50.882435 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65"] Sep 15 11:50:50.882743 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:50.882688 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" podUID="36155bfc-7e1a-495e-a388-ecb25881d37e" containerName="payload-pre-processing" containerID="cri-o://83ade6e1f1e4eebde63752b5a9725d0424ac34cd99b483e62eb4d7bea266e59b" gracePeriod=30 Sep 15 11:50:51.157284 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.157251 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" Sep 15 11:50:51.159987 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.159968 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" Sep 15 11:50:51.274313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.274276 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98727\" (UniqueName: \"kubernetes.io/projected/f13c65d5-c679-49ea-9a9a-0681ab176dea-kube-api-access-98727\") pod \"f13c65d5-c679-49ea-9a9a-0681ab176dea\" (UID: \"f13c65d5-c679-49ea-9a9a-0681ab176dea\") " Sep 15 11:50:51.274449 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.274319 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krxlm\" (UniqueName: \"kubernetes.io/projected/36155bfc-7e1a-495e-a388-ecb25881d37e-kube-api-access-krxlm\") pod \"36155bfc-7e1a-495e-a388-ecb25881d37e\" (UID: \"36155bfc-7e1a-495e-a388-ecb25881d37e\") " Sep 15 11:50:51.274449 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.274383 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13c65d5-c679-49ea-9a9a-0681ab176dea-plugins-config-volume\") pod \"f13c65d5-c679-49ea-9a9a-0681ab176dea\" (UID: \"f13c65d5-c679-49ea-9a9a-0681ab176dea\") " Sep 15 11:50:51.274548 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.274500 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36155bfc-7e1a-495e-a388-ecb25881d37e-plugins-config-volume\") pod \"36155bfc-7e1a-495e-a388-ecb25881d37e\" (UID: \"36155bfc-7e1a-495e-a388-ecb25881d37e\") " Sep 15 11:50:51.274824 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.274795 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f13c65d5-c679-49ea-9a9a-0681ab176dea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f13c65d5-c679-49ea-9a9a-0681ab176dea" (UID: "f13c65d5-c679-49ea-9a9a-0681ab176dea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:50:51.274944 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.274886 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36155bfc-7e1a-495e-a388-ecb25881d37e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36155bfc-7e1a-495e-a388-ecb25881d37e" (UID: "36155bfc-7e1a-495e-a388-ecb25881d37e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:50:51.276488 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.276463 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36155bfc-7e1a-495e-a388-ecb25881d37e-kube-api-access-krxlm" (OuterVolumeSpecName: "kube-api-access-krxlm") pod "36155bfc-7e1a-495e-a388-ecb25881d37e" (UID: "36155bfc-7e1a-495e-a388-ecb25881d37e"). InnerVolumeSpecName "kube-api-access-krxlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:50:51.276488 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.276476 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13c65d5-c679-49ea-9a9a-0681ab176dea-kube-api-access-98727" (OuterVolumeSpecName: "kube-api-access-98727") pod "f13c65d5-c679-49ea-9a9a-0681ab176dea" (UID: "f13c65d5-c679-49ea-9a9a-0681ab176dea"). InnerVolumeSpecName "kube-api-access-98727". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:50:51.375974 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.375932 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36155bfc-7e1a-495e-a388-ecb25881d37e-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:50:51.375974 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.375967 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98727\" (UniqueName: \"kubernetes.io/projected/f13c65d5-c679-49ea-9a9a-0681ab176dea-kube-api-access-98727\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:50:51.375974 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.375980 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krxlm\" (UniqueName: \"kubernetes.io/projected/36155bfc-7e1a-495e-a388-ecb25881d37e-kube-api-access-krxlm\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:50:51.376291 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.375992 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13c65d5-c679-49ea-9a9a-0681ab176dea-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:50:51.888063 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.888009 2573 generic.go:356] "Generic (PLEG): container finished" podID="f13c65d5-c679-49ea-9a9a-0681ab176dea" containerID="2b0e5b52b997d8bbb924f23df454fa1ec7c997003e1ccb19ab278da7f9747c23" exitCode=0 Sep 15 11:50:51.888517 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.888067 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" event={"ID":"f13c65d5-c679-49ea-9a9a-0681ab176dea","Type":"ContainerDied","Data":"2b0e5b52b997d8bbb924f23df454fa1ec7c997003e1ccb19ab278da7f9747c23"} Sep 15 11:50:51.888517 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.888108 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" Sep 15 11:50:51.888517 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.888133 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt" event={"ID":"f13c65d5-c679-49ea-9a9a-0681ab176dea","Type":"ContainerDied","Data":"82cbfc722cdc1cebf012b37a85ae05f4b667c837e1035e0464ac0d838139c99f"} Sep 15 11:50:51.888517 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.888165 2573 scope.go:117] "RemoveContainer" containerID="2b0e5b52b997d8bbb924f23df454fa1ec7c997003e1ccb19ab278da7f9747c23" Sep 15 11:50:51.889948 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.889921 2573 generic.go:356] "Generic (PLEG): container finished" podID="36155bfc-7e1a-495e-a388-ecb25881d37e" containerID="83ade6e1f1e4eebde63752b5a9725d0424ac34cd99b483e62eb4d7bea266e59b" exitCode=0 Sep 15 11:50:51.890097 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.890060 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" event={"ID":"36155bfc-7e1a-495e-a388-ecb25881d37e","Type":"ContainerDied","Data":"83ade6e1f1e4eebde63752b5a9725d0424ac34cd99b483e62eb4d7bea266e59b"} Sep 15 11:50:51.890166 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.890096 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" Sep 15 11:50:51.890166 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.890103 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65" event={"ID":"36155bfc-7e1a-495e-a388-ecb25881d37e","Type":"ContainerDied","Data":"4e1ca4214a9d4a0e11e090f8a25b8335fbbc34c149c3aec4d58d46a4285ed5d5"} Sep 15 11:50:51.901903 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.901881 2573 scope.go:117] "RemoveContainer" containerID="2b0e5b52b997d8bbb924f23df454fa1ec7c997003e1ccb19ab278da7f9747c23" Sep 15 11:50:51.902205 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:51.902185 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0e5b52b997d8bbb924f23df454fa1ec7c997003e1ccb19ab278da7f9747c23\": container with ID starting with 2b0e5b52b997d8bbb924f23df454fa1ec7c997003e1ccb19ab278da7f9747c23 not found: ID does not exist" containerID="2b0e5b52b997d8bbb924f23df454fa1ec7c997003e1ccb19ab278da7f9747c23" Sep 15 11:50:51.902260 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.902215 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0e5b52b997d8bbb924f23df454fa1ec7c997003e1ccb19ab278da7f9747c23"} err="failed to get container status \"2b0e5b52b997d8bbb924f23df454fa1ec7c997003e1ccb19ab278da7f9747c23\": rpc error: code = NotFound desc = could not find container \"2b0e5b52b997d8bbb924f23df454fa1ec7c997003e1ccb19ab278da7f9747c23\": container with ID starting with 2b0e5b52b997d8bbb924f23df454fa1ec7c997003e1ccb19ab278da7f9747c23 not found: ID does not exist" Sep 15 11:50:51.902260 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.902232 2573 scope.go:117] "RemoveContainer" containerID="83ade6e1f1e4eebde63752b5a9725d0424ac34cd99b483e62eb4d7bea266e59b" Sep 15 11:50:51.909943 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.909924 2573 scope.go:117] "RemoveContainer" containerID="83ade6e1f1e4eebde63752b5a9725d0424ac34cd99b483e62eb4d7bea266e59b" Sep 15 11:50:51.910200 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:51.910181 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ade6e1f1e4eebde63752b5a9725d0424ac34cd99b483e62eb4d7bea266e59b\": container with ID starting with 83ade6e1f1e4eebde63752b5a9725d0424ac34cd99b483e62eb4d7bea266e59b not found: ID does not exist" containerID="83ade6e1f1e4eebde63752b5a9725d0424ac34cd99b483e62eb4d7bea266e59b" Sep 15 11:50:51.910265 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.910207 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ade6e1f1e4eebde63752b5a9725d0424ac34cd99b483e62eb4d7bea266e59b"} err="failed to get container status \"83ade6e1f1e4eebde63752b5a9725d0424ac34cd99b483e62eb4d7bea266e59b\": rpc error: code = NotFound desc = could not find container \"83ade6e1f1e4eebde63752b5a9725d0424ac34cd99b483e62eb4d7bea266e59b\": container with ID starting with 83ade6e1f1e4eebde63752b5a9725d0424ac34cd99b483e62eb4d7bea266e59b not found: ID does not exist" Sep 15 11:50:51.913843 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.913823 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt"] Sep 15 11:50:51.917469 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.917449 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ee96ca62-64874d4bfc-tdsjt"] Sep 15 11:50:51.926823 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.926802 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65"] Sep 15 11:50:51.932152 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:51.932132 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee96ca62-857464575d-v7t65"] Sep 15 11:50:53.235222 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:53.235186 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36155bfc-7e1a-495e-a388-ecb25881d37e" path="/var/lib/kubelet/pods/36155bfc-7e1a-495e-a388-ecb25881d37e/volumes" Sep 15 11:50:53.235651 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:53.235636 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f13c65d5-c679-49ea-9a9a-0681ab176dea" path="/var/lib/kubelet/pods/f13c65d5-c679-49ea-9a9a-0681ab176dea/volumes" Sep 15 11:50:57.240654 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.240623 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs"] Sep 15 11:50:57.241124 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.240920 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36155bfc-7e1a-495e-a388-ecb25881d37e" containerName="payload-pre-processing" Sep 15 11:50:57.241124 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.240931 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="36155bfc-7e1a-495e-a388-ecb25881d37e" containerName="payload-pre-processing" Sep 15 11:50:57.241124 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.240967 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f13c65d5-c679-49ea-9a9a-0681ab176dea" containerName="payload-processing" Sep 15 11:50:57.241124 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.240974 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f13c65d5-c679-49ea-9a9a-0681ab176dea" containerName="payload-processing" Sep 15 11:50:57.241124 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.241057 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="36155bfc-7e1a-495e-a388-ecb25881d37e" containerName="payload-pre-processing" Sep 15 11:50:57.241124 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.241066 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f13c65d5-c679-49ea-9a9a-0681ab176dea" containerName="payload-processing" Sep 15 11:50:57.248350 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.248320 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:57.250980 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.250959 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c353d682-serving-cert\"" Sep 15 11:50:57.256060 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.256016 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs"] Sep 15 11:50:57.324334 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.324293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-metrics-certs\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:57.324492 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.324344 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-maas-api-tls\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:57.324492 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.324454 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c07871-2304-4abe-9946-e7b1739af7a2-openshift-service-ca\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:57.324604 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.324513 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tvcq\" (UniqueName: \"kubernetes.io/projected/d8c07871-2304-4abe-9946-e7b1739af7a2-kube-api-access-8tvcq\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:57.425715 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.425682 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c07871-2304-4abe-9946-e7b1739af7a2-openshift-service-ca\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:57.425880 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.425744 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tvcq\" (UniqueName: \"kubernetes.io/projected/d8c07871-2304-4abe-9946-e7b1739af7a2-kube-api-access-8tvcq\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:57.425880 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.425778 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-metrics-certs\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:57.425880 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.425810 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-maas-api-tls\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:57.426089 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:57.425920 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-serving-cert: secret "maas-api-e2e-migrate-c353d682-serving-cert" not found Sep 15 11:50:57.426089 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:57.425992 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-maas-api-tls podName:d8c07871-2304-4abe-9946-e7b1739af7a2 nodeName:}" failed. No retries permitted until 2026-09-15 11:50:57.925969752 +0000 UTC m=+845.155896283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-maas-api-tls") pod "maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" (UID: "d8c07871-2304-4abe-9946-e7b1739af7a2") : secret "maas-api-e2e-migrate-c353d682-serving-cert" not found Sep 15 11:50:57.426459 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.426439 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c07871-2304-4abe-9946-e7b1739af7a2-openshift-service-ca\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:57.428301 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.428283 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-metrics-certs\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:57.436459 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.436431 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tvcq\" (UniqueName: \"kubernetes.io/projected/d8c07871-2304-4abe-9946-e7b1739af7a2-kube-api-access-8tvcq\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:57.929882 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.929847 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-maas-api-tls\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:57.930079 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:57.929997 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-serving-cert: secret "maas-api-e2e-migrate-c353d682-serving-cert" not found Sep 15 11:50:57.930135 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:50:57.930088 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-maas-api-tls podName:d8c07871-2304-4abe-9946-e7b1739af7a2 nodeName:}" failed. No retries permitted until 2026-09-15 11:50:58.930069572 +0000 UTC m=+846.159996102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-maas-api-tls") pod "maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" (UID: "d8c07871-2304-4abe-9946-e7b1739af7a2") : secret "maas-api-e2e-migrate-c353d682-serving-cert" not found Sep 15 11:50:57.954818 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.954784 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w"] Sep 15 11:50:57.958906 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.958886 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" Sep 15 11:50:57.961259 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.961238 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c353d682-dockercfg-qwgnb\"" Sep 15 11:50:57.961346 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.961294 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c353d682\"" Sep 15 11:50:57.966581 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:57.966550 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w"] Sep 15 11:50:58.131667 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.131634 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r6l7\" (UniqueName: \"kubernetes.io/projected/b6d865b3-67d9-4b6b-870f-f64f8333283b-kube-api-access-2r6l7\") pod \"payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w\" (UID: \"b6d865b3-67d9-4b6b-870f-f64f8333283b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" Sep 15 11:50:58.131835 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.131699 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d865b3-67d9-4b6b-870f-f64f8333283b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w\" (UID: \"b6d865b3-67d9-4b6b-870f-f64f8333283b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" Sep 15 11:50:58.232866 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.232782 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r6l7\" (UniqueName: \"kubernetes.io/projected/b6d865b3-67d9-4b6b-870f-f64f8333283b-kube-api-access-2r6l7\") pod \"payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w\" (UID: \"b6d865b3-67d9-4b6b-870f-f64f8333283b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" Sep 15 11:50:58.232866 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.232848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d865b3-67d9-4b6b-870f-f64f8333283b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w\" (UID: \"b6d865b3-67d9-4b6b-870f-f64f8333283b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" Sep 15 11:50:58.233542 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.233522 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d865b3-67d9-4b6b-870f-f64f8333283b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w\" (UID: \"b6d865b3-67d9-4b6b-870f-f64f8333283b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" Sep 15 11:50:58.241150 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.241132 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r6l7\" (UniqueName: \"kubernetes.io/projected/b6d865b3-67d9-4b6b-870f-f64f8333283b-kube-api-access-2r6l7\") pod \"payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w\" (UID: \"b6d865b3-67d9-4b6b-870f-f64f8333283b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" Sep 15 11:50:58.270241 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.270214 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" Sep 15 11:50:58.541105 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.541079 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w"] Sep 15 11:50:58.543681 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:50:58.543654 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d865b3_67d9_4b6b_870f_f64f8333283b.slice/crio-77e4bcad1950ba1ac015a2761b32f487ffb725731e36892c27391e10c854df9a WatchSource:0}: Error finding container 77e4bcad1950ba1ac015a2761b32f487ffb725731e36892c27391e10c854df9a: Status 404 returned error can't find the container with id 77e4bcad1950ba1ac015a2761b32f487ffb725731e36892c27391e10c854df9a Sep 15 11:50:58.925702 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.925617 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" event={"ID":"b6d865b3-67d9-4b6b-870f-f64f8333283b","Type":"ContainerStarted","Data":"0a88138075191f7f7395de5f1ba0807b908680c572bd2f7e9ccfb8d1d2adf9c2"} Sep 15 11:50:58.925702 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.925671 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" event={"ID":"b6d865b3-67d9-4b6b-870f-f64f8333283b","Type":"ContainerStarted","Data":"77e4bcad1950ba1ac015a2761b32f487ffb725731e36892c27391e10c854df9a"} Sep 15 11:50:58.925872 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.925704 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" Sep 15 11:50:58.939321 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.939295 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-maas-api-tls\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:58.941083 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.940954 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" podStartSLOduration=1.9409357919999999 podStartE2EDuration="1.940935792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:50:58.940600631 +0000 UTC m=+846.170527179" watchObservedRunningTime="2026-09-15 11:50:58.940935792 +0000 UTC m=+846.170862330" Sep 15 11:50:58.942169 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:58.942144 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-maas-api-tls\") pod \"maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:59.062645 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:59.062610 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:59.335784 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:59.335757 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs"] Sep 15 11:50:59.337644 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:50:59.337614 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c07871_2304_4abe_9946_e7b1739af7a2.slice/crio-f6e1c557361aa4f8aa0e8ffd4f2531a870ab889cb4e1d340b816e579fb3b4777 WatchSource:0}: Error finding container f6e1c557361aa4f8aa0e8ffd4f2531a870ab889cb4e1d340b816e579fb3b4777: Status 404 returned error can't find the container with id f6e1c557361aa4f8aa0e8ffd4f2531a870ab889cb4e1d340b816e579fb3b4777 Sep 15 11:50:59.932351 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:59.932306 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" event={"ID":"d8c07871-2304-4abe-9946-e7b1739af7a2","Type":"ContainerStarted","Data":"1745a62b8149944f1a2ee28d28201b2895ce234252f7babcbdc0c73b35336048"} Sep 15 11:50:59.933448 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:59.932718 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" event={"ID":"d8c07871-2304-4abe-9946-e7b1739af7a2","Type":"ContainerStarted","Data":"f6e1c557361aa4f8aa0e8ffd4f2531a870ab889cb4e1d340b816e579fb3b4777"} Sep 15 11:50:59.933448 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:59.933418 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:50:59.954917 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:50:59.954856 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" podStartSLOduration=2.5808533000000002 podStartE2EDuration="2.954840657s" totalImagesPullingTime="373.987357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:50:59.953404134 +0000 UTC m=+847.183330675" watchObservedRunningTime="2026-09-15 11:50:59.954840657 +0000 UTC m=+847.184767192" Sep 15 11:51:06.947276 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:06.947248 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:51:09.934635 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:09.934602 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" Sep 15 11:51:27.179558 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.179522 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs"] Sep 15 11:51:27.180056 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.179781 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" podUID="d8c07871-2304-4abe-9946-e7b1739af7a2" containerName="maas-api" containerID="cri-o://1745a62b8149944f1a2ee28d28201b2895ce234252f7babcbdc0c73b35336048" gracePeriod=30 Sep 15 11:51:27.287692 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.287654 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w"] Sep 15 11:51:27.288084 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.288017 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" podUID="b6d865b3-67d9-4b6b-870f-f64f8333283b" containerName="payload-pre-processing" containerID="cri-o://0a88138075191f7f7395de5f1ba0807b908680c572bd2f7e9ccfb8d1d2adf9c2" gracePeriod=30 Sep 15 11:51:27.437756 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.437695 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:51:27.501354 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.501310 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c07871-2304-4abe-9946-e7b1739af7a2-openshift-service-ca\") pod \"d8c07871-2304-4abe-9946-e7b1739af7a2\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " Sep 15 11:51:27.501523 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.501368 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-metrics-certs\") pod \"d8c07871-2304-4abe-9946-e7b1739af7a2\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " Sep 15 11:51:27.501523 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.501441 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-maas-api-tls\") pod \"d8c07871-2304-4abe-9946-e7b1739af7a2\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " Sep 15 11:51:27.501523 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.501493 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tvcq\" (UniqueName: \"kubernetes.io/projected/d8c07871-2304-4abe-9946-e7b1739af7a2-kube-api-access-8tvcq\") pod \"d8c07871-2304-4abe-9946-e7b1739af7a2\" (UID: \"d8c07871-2304-4abe-9946-e7b1739af7a2\") " Sep 15 11:51:27.501698 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.501664 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c07871-2304-4abe-9946-e7b1739af7a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d8c07871-2304-4abe-9946-e7b1739af7a2" (UID: "d8c07871-2304-4abe-9946-e7b1739af7a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:51:27.501813 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.501792 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c07871-2304-4abe-9946-e7b1739af7a2-openshift-service-ca\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:51:27.504572 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.504546 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c07871-2304-4abe-9946-e7b1739af7a2-kube-api-access-8tvcq" (OuterVolumeSpecName: "kube-api-access-8tvcq") pod "d8c07871-2304-4abe-9946-e7b1739af7a2" (UID: "d8c07871-2304-4abe-9946-e7b1739af7a2"). InnerVolumeSpecName "kube-api-access-8tvcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:51:27.505880 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.505854 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d8c07871-2304-4abe-9946-e7b1739af7a2" (UID: "d8c07871-2304-4abe-9946-e7b1739af7a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:51:27.505952 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.505887 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d8c07871-2304-4abe-9946-e7b1739af7a2" (UID: "d8c07871-2304-4abe-9946-e7b1739af7a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:51:27.530650 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.530629 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" Sep 15 11:51:27.602604 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.602571 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r6l7\" (UniqueName: \"kubernetes.io/projected/b6d865b3-67d9-4b6b-870f-f64f8333283b-kube-api-access-2r6l7\") pod \"b6d865b3-67d9-4b6b-870f-f64f8333283b\" (UID: \"b6d865b3-67d9-4b6b-870f-f64f8333283b\") " Sep 15 11:51:27.602772 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.602621 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d865b3-67d9-4b6b-870f-f64f8333283b-plugins-config-volume\") pod \"b6d865b3-67d9-4b6b-870f-f64f8333283b\" (UID: \"b6d865b3-67d9-4b6b-870f-f64f8333283b\") " Sep 15 11:51:27.602823 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.602802 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-maas-api-tls\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:51:27.602823 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.602815 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tvcq\" (UniqueName: \"kubernetes.io/projected/d8c07871-2304-4abe-9946-e7b1739af7a2-kube-api-access-8tvcq\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:51:27.602881 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.602830 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c07871-2304-4abe-9946-e7b1739af7a2-metrics-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:51:27.603127 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.603094 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d865b3-67d9-4b6b-870f-f64f8333283b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6d865b3-67d9-4b6b-870f-f64f8333283b" (UID: "b6d865b3-67d9-4b6b-870f-f64f8333283b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:51:27.605750 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.605721 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d865b3-67d9-4b6b-870f-f64f8333283b-kube-api-access-2r6l7" (OuterVolumeSpecName: "kube-api-access-2r6l7") pod "b6d865b3-67d9-4b6b-870f-f64f8333283b" (UID: "b6d865b3-67d9-4b6b-870f-f64f8333283b"). InnerVolumeSpecName "kube-api-access-2r6l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:51:27.703490 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.703397 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d865b3-67d9-4b6b-870f-f64f8333283b-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:51:27.703490 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:27.703431 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r6l7\" (UniqueName: \"kubernetes.io/projected/b6d865b3-67d9-4b6b-870f-f64f8333283b-kube-api-access-2r6l7\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:51:28.059634 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.059598 2573 generic.go:356] "Generic (PLEG): container finished" podID="b6d865b3-67d9-4b6b-870f-f64f8333283b" containerID="0a88138075191f7f7395de5f1ba0807b908680c572bd2f7e9ccfb8d1d2adf9c2" exitCode=0 Sep 15 11:51:28.059858 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.059651 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" Sep 15 11:51:28.059858 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.059683 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" event={"ID":"b6d865b3-67d9-4b6b-870f-f64f8333283b","Type":"ContainerDied","Data":"0a88138075191f7f7395de5f1ba0807b908680c572bd2f7e9ccfb8d1d2adf9c2"} Sep 15 11:51:28.059858 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.059740 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w" event={"ID":"b6d865b3-67d9-4b6b-870f-f64f8333283b","Type":"ContainerDied","Data":"77e4bcad1950ba1ac015a2761b32f487ffb725731e36892c27391e10c854df9a"} Sep 15 11:51:28.059858 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.059772 2573 scope.go:117] "RemoveContainer" containerID="0a88138075191f7f7395de5f1ba0807b908680c572bd2f7e9ccfb8d1d2adf9c2" Sep 15 11:51:28.061613 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.061146 2573 generic.go:356] "Generic (PLEG): container finished" podID="d8c07871-2304-4abe-9946-e7b1739af7a2" containerID="1745a62b8149944f1a2ee28d28201b2895ce234252f7babcbdc0c73b35336048" exitCode=0 Sep 15 11:51:28.061613 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.061173 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" event={"ID":"d8c07871-2304-4abe-9946-e7b1739af7a2","Type":"ContainerDied","Data":"1745a62b8149944f1a2ee28d28201b2895ce234252f7babcbdc0c73b35336048"} Sep 15 11:51:28.061613 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.061216 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" Sep 15 11:51:28.061613 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.061228 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs" event={"ID":"d8c07871-2304-4abe-9946-e7b1739af7a2","Type":"ContainerDied","Data":"f6e1c557361aa4f8aa0e8ffd4f2531a870ab889cb4e1d340b816e579fb3b4777"} Sep 15 11:51:28.068367 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.068346 2573 scope.go:117] "RemoveContainer" containerID="0a88138075191f7f7395de5f1ba0807b908680c572bd2f7e9ccfb8d1d2adf9c2" Sep 15 11:51:28.068628 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:51:28.068609 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a88138075191f7f7395de5f1ba0807b908680c572bd2f7e9ccfb8d1d2adf9c2\": container with ID starting with 0a88138075191f7f7395de5f1ba0807b908680c572bd2f7e9ccfb8d1d2adf9c2 not found: ID does not exist" containerID="0a88138075191f7f7395de5f1ba0807b908680c572bd2f7e9ccfb8d1d2adf9c2" Sep 15 11:51:28.068716 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.068633 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a88138075191f7f7395de5f1ba0807b908680c572bd2f7e9ccfb8d1d2adf9c2"} err="failed to get container status \"0a88138075191f7f7395de5f1ba0807b908680c572bd2f7e9ccfb8d1d2adf9c2\": rpc error: code = NotFound desc = could not find container \"0a88138075191f7f7395de5f1ba0807b908680c572bd2f7e9ccfb8d1d2adf9c2\": container with ID starting with 0a88138075191f7f7395de5f1ba0807b908680c572bd2f7e9ccfb8d1d2adf9c2 not found: ID does not exist" Sep 15 11:51:28.068716 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.068648 2573 scope.go:117] "RemoveContainer" containerID="1745a62b8149944f1a2ee28d28201b2895ce234252f7babcbdc0c73b35336048" Sep 15 11:51:28.077517 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.077499 2573 scope.go:117] "RemoveContainer" containerID="1745a62b8149944f1a2ee28d28201b2895ce234252f7babcbdc0c73b35336048" Sep 15 11:51:28.077786 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:51:28.077762 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1745a62b8149944f1a2ee28d28201b2895ce234252f7babcbdc0c73b35336048\": container with ID starting with 1745a62b8149944f1a2ee28d28201b2895ce234252f7babcbdc0c73b35336048 not found: ID does not exist" containerID="1745a62b8149944f1a2ee28d28201b2895ce234252f7babcbdc0c73b35336048" Sep 15 11:51:28.077843 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.077792 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1745a62b8149944f1a2ee28d28201b2895ce234252f7babcbdc0c73b35336048"} err="failed to get container status \"1745a62b8149944f1a2ee28d28201b2895ce234252f7babcbdc0c73b35336048\": rpc error: code = NotFound desc = could not find container \"1745a62b8149944f1a2ee28d28201b2895ce234252f7babcbdc0c73b35336048\": container with ID starting with 1745a62b8149944f1a2ee28d28201b2895ce234252f7babcbdc0c73b35336048 not found: ID does not exist" Sep 15 11:51:28.104475 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.104452 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w"] Sep 15 11:51:28.116129 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.116108 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c353d682-556fb99bf4-6nl9w"] Sep 15 11:51:28.132243 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.132222 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs"] Sep 15 11:51:28.143386 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:28.143363 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c353d682-778b4cb9-bj7gs"] Sep 15 11:51:29.234741 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.234709 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6d865b3-67d9-4b6b-870f-f64f8333283b" path="/var/lib/kubelet/pods/b6d865b3-67d9-4b6b-870f-f64f8333283b/volumes" Sep 15 11:51:29.235118 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.235109 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8c07871-2304-4abe-9946-e7b1739af7a2" path="/var/lib/kubelet/pods/d8c07871-2304-4abe-9946-e7b1739af7a2/volumes" Sep 15 11:51:29.356630 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.356597 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8"] Sep 15 11:51:29.356979 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.356963 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c07871-2304-4abe-9946-e7b1739af7a2" containerName="maas-api" Sep 15 11:51:29.357088 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.356982 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c07871-2304-4abe-9946-e7b1739af7a2" containerName="maas-api" Sep 15 11:51:29.357088 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.357044 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d865b3-67d9-4b6b-870f-f64f8333283b" containerName="payload-pre-processing" Sep 15 11:51:29.357088 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.357054 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d865b3-67d9-4b6b-870f-f64f8333283b" containerName="payload-pre-processing" Sep 15 11:51:29.357240 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.357138 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d865b3-67d9-4b6b-870f-f64f8333283b" containerName="payload-pre-processing" Sep 15 11:51:29.357240 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.357149 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c07871-2304-4abe-9946-e7b1739af7a2" containerName="maas-api" Sep 15 11:51:29.361628 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.361608 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.364897 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.364875 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-155ffe\"/\"test-model-155ffe-kserve-self-signed-certs\"" Sep 15 11:51:29.365015 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.364897 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-155ffe\"/\"openshift-service-ca.crt\"" Sep 15 11:51:29.365015 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.364897 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-155ffe\"/\"default-dockercfg-gxpx5\"" Sep 15 11:51:29.365015 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.364886 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-155ffe\"/\"kube-root-ca.crt\"" Sep 15 11:51:29.370538 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.370512 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8"] Sep 15 11:51:29.416870 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.416849 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-tmp-dir\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.417013 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.416895 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-dshm\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.417013 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.416926 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phgbh\" (UniqueName: \"kubernetes.io/projected/87bc530c-da3b-4502-9cb8-6410dc1d0fef-kube-api-access-phgbh\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.417013 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.416974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87bc530c-da3b-4502-9cb8-6410dc1d0fef-tls-certs\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.417013 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.416997 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-home\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.417235 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.417087 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-model-cache\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.518359 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.518323 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-tmp-dir\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.518545 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.518370 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-dshm\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.518545 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.518388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phgbh\" (UniqueName: \"kubernetes.io/projected/87bc530c-da3b-4502-9cb8-6410dc1d0fef-kube-api-access-phgbh\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.518545 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.518504 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87bc530c-da3b-4502-9cb8-6410dc1d0fef-tls-certs\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.518545 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.518540 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-home\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.518777 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.518595 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-model-cache\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.518884 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.518863 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-tmp-dir\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.518950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.518866 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-home\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.519003 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.518956 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-model-cache\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.520750 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.520728 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-dshm\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.521076 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.521056 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87bc530c-da3b-4502-9cb8-6410dc1d0fef-tls-certs\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.529534 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.529510 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phgbh\" (UniqueName: \"kubernetes.io/projected/87bc530c-da3b-4502-9cb8-6410dc1d0fef-kube-api-access-phgbh\") pod \"test-model-155ffe-kserve-6d6bc78465-ghwf8\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.672103 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.672070 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:29.942792 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:29.942761 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8"] Sep 15 11:51:29.944736 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:51:29.944707 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87bc530c_da3b_4502_9cb8_6410dc1d0fef.slice/crio-483b7e0350f7a2cc0e8c387c7ccd182a61bd82124b21ff6ddbd401abbfbe318b WatchSource:0}: Error finding container 483b7e0350f7a2cc0e8c387c7ccd182a61bd82124b21ff6ddbd401abbfbe318b: Status 404 returned error can't find the container with id 483b7e0350f7a2cc0e8c387c7ccd182a61bd82124b21ff6ddbd401abbfbe318b Sep 15 11:51:30.073758 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:30.073725 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" event={"ID":"87bc530c-da3b-4502-9cb8-6410dc1d0fef","Type":"ContainerStarted","Data":"2dfb68232c3a02b0fd4a583c19f03afbe0301bb2e67b9a77b45dd1e57fd6d50f"} Sep 15 11:51:30.073920 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:30.073768 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" event={"ID":"87bc530c-da3b-4502-9cb8-6410dc1d0fef","Type":"ContainerStarted","Data":"483b7e0350f7a2cc0e8c387c7ccd182a61bd82124b21ff6ddbd401abbfbe318b"} Sep 15 11:51:30.095056 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:30.094994 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" podStartSLOduration=1.094975264 podStartE2EDuration="1.094975264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:51:30.094841028 +0000 UTC m=+877.324767563" watchObservedRunningTime="2026-09-15 11:51:30.094975264 +0000 UTC m=+877.324901800" Sep 15 11:51:33.174993 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.174956 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv"] Sep 15 11:51:33.178577 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.178556 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.181200 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.181178 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b02eba29-serving-cert\"" Sep 15 11:51:33.189604 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.189578 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv"] Sep 15 11:51:33.255307 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.255273 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2926q\" (UniqueName: \"kubernetes.io/projected/5578f590-384f-4b86-bcf8-7b0147e5f3e7-kube-api-access-2926q\") pod \"maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.255496 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.255312 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5578f590-384f-4b86-bcf8-7b0147e5f3e7-openshift-service-ca\") pod \"maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.255496 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.255411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-maas-api-tls\") pod \"maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.255496 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.255440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-metrics-certs\") pod \"maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.356364 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.356332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2926q\" (UniqueName: \"kubernetes.io/projected/5578f590-384f-4b86-bcf8-7b0147e5f3e7-kube-api-access-2926q\") pod \"maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.356364 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.356369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5578f590-384f-4b86-bcf8-7b0147e5f3e7-openshift-service-ca\") pod \"maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.356608 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.356428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-maas-api-tls\") pod \"maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.356608 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.356450 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-metrics-certs\") pod \"maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.356608 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:51:33.356571 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-serving-cert: secret "maas-api-e2e-ait-b02eba29-serving-cert" not found Sep 15 11:51:33.356777 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:51:33.356632 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-maas-api-tls podName:5578f590-384f-4b86-bcf8-7b0147e5f3e7 nodeName:}" failed. No retries permitted until 2026-09-15 11:51:33.856616669 +0000 UTC m=+881.086543187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-maas-api-tls") pod "maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" (UID: "5578f590-384f-4b86-bcf8-7b0147e5f3e7") : secret "maas-api-e2e-ait-b02eba29-serving-cert" not found Sep 15 11:51:33.357160 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.357137 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5578f590-384f-4b86-bcf8-7b0147e5f3e7-openshift-service-ca\") pod \"maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.358843 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.358824 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-metrics-certs\") pod \"maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.364109 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.364086 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2926q\" (UniqueName: \"kubernetes.io/projected/5578f590-384f-4b86-bcf8-7b0147e5f3e7-kube-api-access-2926q\") pod \"maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.772686 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.772654 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq"] Sep 15 11:51:33.776200 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.776179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" Sep 15 11:51:33.778483 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.778466 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b02eba29-dockercfg-8dhw9\"" Sep 15 11:51:33.778588 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.778466 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b02eba29\"" Sep 15 11:51:33.786080 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.786054 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq"] Sep 15 11:51:33.813764 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.813730 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn"] Sep 15 11:51:33.816839 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.816823 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" Sep 15 11:51:33.826236 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.826212 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn"] Sep 15 11:51:33.861084 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.861059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6eb202-66d8-4539-9515-7d68f83fb14f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq\" (UID: \"5e6eb202-66d8-4539-9515-7d68f83fb14f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" Sep 15 11:51:33.861206 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.861140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjkjl\" (UniqueName: \"kubernetes.io/projected/5e6eb202-66d8-4539-9515-7d68f83fb14f-kube-api-access-mjkjl\") pod \"payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq\" (UID: \"5e6eb202-66d8-4539-9515-7d68f83fb14f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" Sep 15 11:51:33.861206 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.861177 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-maas-api-tls\") pod \"maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.863435 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.863410 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-maas-api-tls\") pod \"maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:33.962353 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.962314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w9nv\" (UniqueName: \"kubernetes.io/projected/18cde059-c1f1-4af0-8c25-62ef4e37a9fd-kube-api-access-4w9nv\") pod \"payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn\" (UID: \"18cde059-c1f1-4af0-8c25-62ef4e37a9fd\") " pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" Sep 15 11:51:33.962496 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.962376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6eb202-66d8-4539-9515-7d68f83fb14f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq\" (UID: \"5e6eb202-66d8-4539-9515-7d68f83fb14f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" Sep 15 11:51:33.962496 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.962469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjkjl\" (UniqueName: \"kubernetes.io/projected/5e6eb202-66d8-4539-9515-7d68f83fb14f-kube-api-access-mjkjl\") pod \"payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq\" (UID: \"5e6eb202-66d8-4539-9515-7d68f83fb14f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" Sep 15 11:51:33.962572 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.962507 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18cde059-c1f1-4af0-8c25-62ef4e37a9fd-plugins-config-volume\") pod \"payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn\" (UID: \"18cde059-c1f1-4af0-8c25-62ef4e37a9fd\") " pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" Sep 15 11:51:33.962967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.962951 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6eb202-66d8-4539-9515-7d68f83fb14f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq\" (UID: \"5e6eb202-66d8-4539-9515-7d68f83fb14f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" Sep 15 11:51:33.970343 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:33.970316 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjkjl\" (UniqueName: \"kubernetes.io/projected/5e6eb202-66d8-4539-9515-7d68f83fb14f-kube-api-access-mjkjl\") pod \"payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq\" (UID: \"5e6eb202-66d8-4539-9515-7d68f83fb14f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" Sep 15 11:51:34.063363 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:34.063290 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18cde059-c1f1-4af0-8c25-62ef4e37a9fd-plugins-config-volume\") pod \"payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn\" (UID: \"18cde059-c1f1-4af0-8c25-62ef4e37a9fd\") " pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" Sep 15 11:51:34.063363 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:34.063332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w9nv\" (UniqueName: \"kubernetes.io/projected/18cde059-c1f1-4af0-8c25-62ef4e37a9fd-kube-api-access-4w9nv\") pod \"payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn\" (UID: \"18cde059-c1f1-4af0-8c25-62ef4e37a9fd\") " pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" Sep 15 11:51:34.063932 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:34.063913 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18cde059-c1f1-4af0-8c25-62ef4e37a9fd-plugins-config-volume\") pod \"payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn\" (UID: \"18cde059-c1f1-4af0-8c25-62ef4e37a9fd\") " pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" Sep 15 11:51:34.071528 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:34.071503 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w9nv\" (UniqueName: \"kubernetes.io/projected/18cde059-c1f1-4af0-8c25-62ef4e37a9fd-kube-api-access-4w9nv\") pod \"payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn\" (UID: \"18cde059-c1f1-4af0-8c25-62ef4e37a9fd\") " pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" Sep 15 11:51:34.087335 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:34.087312 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" Sep 15 11:51:34.090225 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:34.090205 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:34.129228 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:34.129193 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" Sep 15 11:51:34.364839 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:34.364812 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq"] Sep 15 11:51:34.367705 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:51:34.367677 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6eb202_66d8_4539_9515_7d68f83fb14f.slice/crio-98eb2783230a88bfc73bcb1b517b4463e9280514c653d2d0ea04fa9af344822f WatchSource:0}: Error finding container 98eb2783230a88bfc73bcb1b517b4463e9280514c653d2d0ea04fa9af344822f: Status 404 returned error can't find the container with id 98eb2783230a88bfc73bcb1b517b4463e9280514c653d2d0ea04fa9af344822f Sep 15 11:51:34.368136 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:34.368071 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv"] Sep 15 11:51:34.370626 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:51:34.370520 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5578f590_384f_4b86_bcf8_7b0147e5f3e7.slice/crio-7ea7115e6c486a7d7e39a442b315395f67e73d9fe056cb49958e5e3a154b5f34 WatchSource:0}: Error finding container 7ea7115e6c486a7d7e39a442b315395f67e73d9fe056cb49958e5e3a154b5f34: Status 404 returned error can't find the container with id 7ea7115e6c486a7d7e39a442b315395f67e73d9fe056cb49958e5e3a154b5f34 Sep 15 11:51:34.431446 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:34.431423 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn"] Sep 15 11:51:34.433005 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:51:34.432982 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cde059_c1f1_4af0_8c25_62ef4e37a9fd.slice/crio-91b3ce228da444f1f025c0b6cb7bf8d422ec2260731d1ca136f5acadfd895879 WatchSource:0}: Error finding container 91b3ce228da444f1f025c0b6cb7bf8d422ec2260731d1ca136f5acadfd895879: Status 404 returned error can't find the container with id 91b3ce228da444f1f025c0b6cb7bf8d422ec2260731d1ca136f5acadfd895879 Sep 15 11:51:35.098938 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:35.098852 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" event={"ID":"5578f590-384f-4b86-bcf8-7b0147e5f3e7","Type":"ContainerStarted","Data":"972394e9d5ffdcc4745ccea4e87bba2eceef45f19c7c6d7ce8265a8de92fa490"} Sep 15 11:51:35.099087 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:35.098953 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" event={"ID":"5578f590-384f-4b86-bcf8-7b0147e5f3e7","Type":"ContainerStarted","Data":"7ea7115e6c486a7d7e39a442b315395f67e73d9fe056cb49958e5e3a154b5f34"} Sep 15 11:51:35.099087 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:35.098983 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:35.100845 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:35.100817 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" event={"ID":"5e6eb202-66d8-4539-9515-7d68f83fb14f","Type":"ContainerStarted","Data":"1289a5fe6304f0856d9cd30b741bf92fcc49a48bf328ff1dd30cf40a60bd175f"} Sep 15 11:51:35.100995 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:35.100867 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" event={"ID":"5e6eb202-66d8-4539-9515-7d68f83fb14f","Type":"ContainerStarted","Data":"98eb2783230a88bfc73bcb1b517b4463e9280514c653d2d0ea04fa9af344822f"} Sep 15 11:51:35.100995 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:35.100918 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" Sep 15 11:51:35.102558 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:35.102534 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" event={"ID":"18cde059-c1f1-4af0-8c25-62ef4e37a9fd","Type":"ContainerStarted","Data":"be75e048171452d685773b8e830d589e1264e68dcdefa6158281cdc4587d72c7"} Sep 15 11:51:35.102679 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:35.102579 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" event={"ID":"18cde059-c1f1-4af0-8c25-62ef4e37a9fd","Type":"ContainerStarted","Data":"91b3ce228da444f1f025c0b6cb7bf8d422ec2260731d1ca136f5acadfd895879"} Sep 15 11:51:35.102679 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:35.102607 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" Sep 15 11:51:35.120181 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:35.120135 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" podStartSLOduration=1.7097465120000002 podStartE2EDuration="2.120122613s" totalImagesPullingTime="410.376101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:51:35.118305153 +0000 UTC m=+882.348231689" watchObservedRunningTime="2026-09-15 11:51:35.120122613 +0000 UTC m=+882.350049149" Sep 15 11:51:35.136598 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:35.136547 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" podStartSLOduration=2.136530545 podStartE2EDuration="2.136530545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:51:35.134834571 +0000 UTC m=+882.364761108" watchObservedRunningTime="2026-09-15 11:51:35.136530545 +0000 UTC m=+882.366457085" Sep 15 11:51:35.153352 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:35.153302 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" podStartSLOduration=2.153288734 podStartE2EDuration="2.153288734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:51:35.151963398 +0000 UTC m=+882.381889932" watchObservedRunningTime="2026-09-15 11:51:35.153288734 +0000 UTC m=+882.383215343" Sep 15 11:51:39.673271 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:39.673229 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:39.673651 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:39.673289 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:39.685751 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:39.685719 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:40.137010 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:40.136981 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:51:41.112807 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:41.112733 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:51:46.109842 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:46.109765 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" Sep 15 11:51:46.110240 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:46.109850 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" Sep 15 11:51:49.574526 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.574487 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn"] Sep 15 11:51:49.578040 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.578006 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.581398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.581347 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1b147a\"/\"openshift-service-ca.crt\"" Sep 15 11:51:49.581398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.581358 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1b147a\"/\"kube-root-ca.crt\"" Sep 15 11:51:49.581398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.581359 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1b147a\"/\"default-dockercfg-55pr4\"" Sep 15 11:51:49.581727 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.581701 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1b147a\"/\"test-model-1b147a-kserve-self-signed-certs\"" Sep 15 11:51:49.587352 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.587327 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn"] Sep 15 11:51:49.698967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.698935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-dshm\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.698967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.698974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-model-cache\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.699165 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.698993 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-home\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.699165 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.699067 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-tmp-dir\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.699165 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.699119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ace007a-41eb-4344-af2f-3cd62646196a-tls-certs\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.699165 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.699146 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzhkq\" (UniqueName: \"kubernetes.io/projected/7ace007a-41eb-4344-af2f-3cd62646196a-kube-api-access-zzhkq\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.800448 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.800413 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-dshm\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.800603 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.800454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-model-cache\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.800603 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.800570 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-home\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.800716 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.800608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-tmp-dir\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.800716 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.800633 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ace007a-41eb-4344-af2f-3cd62646196a-tls-certs\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.800716 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.800678 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzhkq\" (UniqueName: \"kubernetes.io/projected/7ace007a-41eb-4344-af2f-3cd62646196a-kube-api-access-zzhkq\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.800864 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.800812 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-model-cache\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.801008 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.800985 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-home\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.801106 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.800993 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-tmp-dir\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.802878 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.802849 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-dshm\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.802981 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.802963 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ace007a-41eb-4344-af2f-3cd62646196a-tls-certs\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.811817 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.811796 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzhkq\" (UniqueName: \"kubernetes.io/projected/7ace007a-41eb-4344-af2f-3cd62646196a-kube-api-access-zzhkq\") pod \"test-model-1b147a-kserve-7c99979865-rjcgn\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:49.888921 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:49.888858 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:50.159555 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:50.159526 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn"] Sep 15 11:51:50.160815 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:51:50.160786 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ace007a_41eb_4344_af2f_3cd62646196a.slice/crio-64257b92b9236f4f33a37acbbdda49dc6dde2f54f023cd018d2ba4b9412437b2 WatchSource:0}: Error finding container 64257b92b9236f4f33a37acbbdda49dc6dde2f54f023cd018d2ba4b9412437b2: Status 404 returned error can't find the container with id 64257b92b9236f4f33a37acbbdda49dc6dde2f54f023cd018d2ba4b9412437b2 Sep 15 11:51:50.177135 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:50.177104 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" event={"ID":"7ace007a-41eb-4344-af2f-3cd62646196a","Type":"ContainerStarted","Data":"64257b92b9236f4f33a37acbbdda49dc6dde2f54f023cd018d2ba4b9412437b2"} Sep 15 11:51:51.183673 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:51.183628 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" event={"ID":"7ace007a-41eb-4344-af2f-3cd62646196a","Type":"ContainerStarted","Data":"265c0015245886566367680f888e9ebcd3987e4a93dbded496b13629ecc0240e"} Sep 15 11:51:51.203516 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:51.203467 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" podStartSLOduration=2.203450698 podStartE2EDuration="2.203450698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:51:51.201344734 +0000 UTC m=+898.431271282" watchObservedRunningTime="2026-09-15 11:51:51.203450698 +0000 UTC m=+898.433377235" Sep 15 11:51:53.239898 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:53.239865 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 11:51:53.240714 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:53.240692 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 11:51:53.247376 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:53.247360 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 11:51:53.248646 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:53.248628 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 11:51:59.889709 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:59.889663 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:59.892367 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:59.889723 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:51:59.902307 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:51:59.902276 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:52:00.237368 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:00.237287 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:52:13.097683 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.097645 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv"] Sep 15 11:52:13.098146 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.097863 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" podUID="5578f590-384f-4b86-bcf8-7b0147e5f3e7" containerName="maas-api" containerID="cri-o://972394e9d5ffdcc4745ccea4e87bba2eceef45f19c7c6d7ce8265a8de92fa490" gracePeriod=30 Sep 15 11:52:13.156557 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.156522 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn"] Sep 15 11:52:13.156823 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.156741 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" podUID="18cde059-c1f1-4af0-8c25-62ef4e37a9fd" containerName="payload-processing" containerID="cri-o://be75e048171452d685773b8e830d589e1264e68dcdefa6158281cdc4587d72c7" gracePeriod=30 Sep 15 11:52:13.158471 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.158448 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq"] Sep 15 11:52:13.158672 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.158650 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" podUID="5e6eb202-66d8-4539-9515-7d68f83fb14f" containerName="payload-pre-processing" containerID="cri-o://1289a5fe6304f0856d9cd30b741bf92fcc49a48bf328ff1dd30cf40a60bd175f" gracePeriod=30 Sep 15 11:52:13.287544 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.287491 2573 generic.go:356] "Generic (PLEG): container finished" podID="5e6eb202-66d8-4539-9515-7d68f83fb14f" containerID="1289a5fe6304f0856d9cd30b741bf92fcc49a48bf328ff1dd30cf40a60bd175f" exitCode=0 Sep 15 11:52:13.287544 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.287525 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" event={"ID":"5e6eb202-66d8-4539-9515-7d68f83fb14f","Type":"ContainerDied","Data":"1289a5fe6304f0856d9cd30b741bf92fcc49a48bf328ff1dd30cf40a60bd175f"} Sep 15 11:52:13.289524 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.289499 2573 generic.go:356] "Generic (PLEG): container finished" podID="18cde059-c1f1-4af0-8c25-62ef4e37a9fd" containerID="be75e048171452d685773b8e830d589e1264e68dcdefa6158281cdc4587d72c7" exitCode=0 Sep 15 11:52:13.289656 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.289585 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" event={"ID":"18cde059-c1f1-4af0-8c25-62ef4e37a9fd","Type":"ContainerDied","Data":"be75e048171452d685773b8e830d589e1264e68dcdefa6158281cdc4587d72c7"} Sep 15 11:52:13.291869 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.291840 2573 generic.go:356] "Generic (PLEG): container finished" podID="5578f590-384f-4b86-bcf8-7b0147e5f3e7" containerID="972394e9d5ffdcc4745ccea4e87bba2eceef45f19c7c6d7ce8265a8de92fa490" exitCode=0 Sep 15 11:52:13.291971 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.291879 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" event={"ID":"5578f590-384f-4b86-bcf8-7b0147e5f3e7","Type":"ContainerDied","Data":"972394e9d5ffdcc4745ccea4e87bba2eceef45f19c7c6d7ce8265a8de92fa490"} Sep 15 11:52:13.405620 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.405586 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:52:13.457016 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.456991 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" Sep 15 11:52:13.460107 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.460087 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" Sep 15 11:52:13.508829 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.508797 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2926q\" (UniqueName: \"kubernetes.io/projected/5578f590-384f-4b86-bcf8-7b0147e5f3e7-kube-api-access-2926q\") pod \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " Sep 15 11:52:13.508992 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.508861 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-metrics-certs\") pod \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " Sep 15 11:52:13.509069 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.508991 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5578f590-384f-4b86-bcf8-7b0147e5f3e7-openshift-service-ca\") pod \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " Sep 15 11:52:13.509069 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.509044 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-maas-api-tls\") pod \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\" (UID: \"5578f590-384f-4b86-bcf8-7b0147e5f3e7\") " Sep 15 11:52:13.509373 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.509078 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w9nv\" (UniqueName: \"kubernetes.io/projected/18cde059-c1f1-4af0-8c25-62ef4e37a9fd-kube-api-access-4w9nv\") pod \"18cde059-c1f1-4af0-8c25-62ef4e37a9fd\" (UID: \"18cde059-c1f1-4af0-8c25-62ef4e37a9fd\") " Sep 15 11:52:13.509373 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.509114 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6eb202-66d8-4539-9515-7d68f83fb14f-plugins-config-volume\") pod \"5e6eb202-66d8-4539-9515-7d68f83fb14f\" (UID: \"5e6eb202-66d8-4539-9515-7d68f83fb14f\") " Sep 15 11:52:13.509373 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.509141 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18cde059-c1f1-4af0-8c25-62ef4e37a9fd-plugins-config-volume\") pod \"18cde059-c1f1-4af0-8c25-62ef4e37a9fd\" (UID: \"18cde059-c1f1-4af0-8c25-62ef4e37a9fd\") " Sep 15 11:52:13.509373 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.509165 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjkjl\" (UniqueName: \"kubernetes.io/projected/5e6eb202-66d8-4539-9515-7d68f83fb14f-kube-api-access-mjkjl\") pod \"5e6eb202-66d8-4539-9515-7d68f83fb14f\" (UID: \"5e6eb202-66d8-4539-9515-7d68f83fb14f\") " Sep 15 11:52:13.509596 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.509436 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5578f590-384f-4b86-bcf8-7b0147e5f3e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5578f590-384f-4b86-bcf8-7b0147e5f3e7" (UID: "5578f590-384f-4b86-bcf8-7b0147e5f3e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:52:13.509596 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.509498 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6eb202-66d8-4539-9515-7d68f83fb14f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e6eb202-66d8-4539-9515-7d68f83fb14f" (UID: "5e6eb202-66d8-4539-9515-7d68f83fb14f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:52:13.509833 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.509810 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cde059-c1f1-4af0-8c25-62ef4e37a9fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "18cde059-c1f1-4af0-8c25-62ef4e37a9fd" (UID: "18cde059-c1f1-4af0-8c25-62ef4e37a9fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:52:13.511458 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.511438 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6eb202-66d8-4539-9515-7d68f83fb14f-kube-api-access-mjkjl" (OuterVolumeSpecName: "kube-api-access-mjkjl") pod "5e6eb202-66d8-4539-9515-7d68f83fb14f" (UID: "5e6eb202-66d8-4539-9515-7d68f83fb14f"). InnerVolumeSpecName "kube-api-access-mjkjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:52:13.511647 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.511618 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cde059-c1f1-4af0-8c25-62ef4e37a9fd-kube-api-access-4w9nv" (OuterVolumeSpecName: "kube-api-access-4w9nv") pod "18cde059-c1f1-4af0-8c25-62ef4e37a9fd" (UID: "18cde059-c1f1-4af0-8c25-62ef4e37a9fd"). InnerVolumeSpecName "kube-api-access-4w9nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:52:13.511744 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.511659 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5578f590-384f-4b86-bcf8-7b0147e5f3e7" (UID: "5578f590-384f-4b86-bcf8-7b0147e5f3e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:52:13.511744 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.511675 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5578f590-384f-4b86-bcf8-7b0147e5f3e7-kube-api-access-2926q" (OuterVolumeSpecName: "kube-api-access-2926q") pod "5578f590-384f-4b86-bcf8-7b0147e5f3e7" (UID: "5578f590-384f-4b86-bcf8-7b0147e5f3e7"). InnerVolumeSpecName "kube-api-access-2926q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:52:13.511744 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.511676 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5578f590-384f-4b86-bcf8-7b0147e5f3e7" (UID: "5578f590-384f-4b86-bcf8-7b0147e5f3e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:52:13.610779 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.610743 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6eb202-66d8-4539-9515-7d68f83fb14f-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:13.610779 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.610776 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18cde059-c1f1-4af0-8c25-62ef4e37a9fd-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:13.610779 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.610786 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjkjl\" (UniqueName: \"kubernetes.io/projected/5e6eb202-66d8-4539-9515-7d68f83fb14f-kube-api-access-mjkjl\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:13.611050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.610795 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2926q\" (UniqueName: \"kubernetes.io/projected/5578f590-384f-4b86-bcf8-7b0147e5f3e7-kube-api-access-2926q\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:13.611050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.610805 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-metrics-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:13.611050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.610813 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5578f590-384f-4b86-bcf8-7b0147e5f3e7-openshift-service-ca\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:13.611050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.610821 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5578f590-384f-4b86-bcf8-7b0147e5f3e7-maas-api-tls\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:13.611050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:13.610830 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w9nv\" (UniqueName: \"kubernetes.io/projected/18cde059-c1f1-4af0-8c25-62ef4e37a9fd-kube-api-access-4w9nv\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:14.298114 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.298086 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" Sep 15 11:52:14.298114 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.298096 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv" event={"ID":"5578f590-384f-4b86-bcf8-7b0147e5f3e7","Type":"ContainerDied","Data":"7ea7115e6c486a7d7e39a442b315395f67e73d9fe056cb49958e5e3a154b5f34"} Sep 15 11:52:14.298593 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.298181 2573 scope.go:117] "RemoveContainer" containerID="972394e9d5ffdcc4745ccea4e87bba2eceef45f19c7c6d7ce8265a8de92fa490" Sep 15 11:52:14.299667 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.299415 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" Sep 15 11:52:14.299667 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.299436 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq" event={"ID":"5e6eb202-66d8-4539-9515-7d68f83fb14f","Type":"ContainerDied","Data":"98eb2783230a88bfc73bcb1b517b4463e9280514c653d2d0ea04fa9af344822f"} Sep 15 11:52:14.304100 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.304071 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" event={"ID":"18cde059-c1f1-4af0-8c25-62ef4e37a9fd","Type":"ContainerDied","Data":"91b3ce228da444f1f025c0b6cb7bf8d422ec2260731d1ca136f5acadfd895879"} Sep 15 11:52:14.304227 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.304086 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn" Sep 15 11:52:14.309944 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.309919 2573 scope.go:117] "RemoveContainer" containerID="1289a5fe6304f0856d9cd30b741bf92fcc49a48bf328ff1dd30cf40a60bd175f" Sep 15 11:52:14.319049 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.319011 2573 scope.go:117] "RemoveContainer" containerID="be75e048171452d685773b8e830d589e1264e68dcdefa6158281cdc4587d72c7" Sep 15 11:52:14.334399 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.334373 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv"] Sep 15 11:52:14.342396 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.342375 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b02eba29-768c6cdf4c-d7pdv"] Sep 15 11:52:14.356456 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.356434 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq"] Sep 15 11:52:14.362933 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.362881 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b02eba29-b5475f798-wxpfq"] Sep 15 11:52:14.374417 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.374393 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn"] Sep 15 11:52:14.381058 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:14.381038 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b02eba29-745659bb49-tvwfn"] Sep 15 11:52:15.234624 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:15.234588 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18cde059-c1f1-4af0-8c25-62ef4e37a9fd" path="/var/lib/kubelet/pods/18cde059-c1f1-4af0-8c25-62ef4e37a9fd/volumes" Sep 15 11:52:15.234989 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:15.234974 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5578f590-384f-4b86-bcf8-7b0147e5f3e7" path="/var/lib/kubelet/pods/5578f590-384f-4b86-bcf8-7b0147e5f3e7/volumes" Sep 15 11:52:15.235316 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:15.235303 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e6eb202-66d8-4539-9515-7d68f83fb14f" path="/var/lib/kubelet/pods/5e6eb202-66d8-4539-9515-7d68f83fb14f/volumes" Sep 15 11:52:20.994263 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:20.994228 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg"] Sep 15 11:52:20.994956 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:20.994814 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5578f590-384f-4b86-bcf8-7b0147e5f3e7" containerName="maas-api" Sep 15 11:52:20.994956 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:20.994838 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5578f590-384f-4b86-bcf8-7b0147e5f3e7" containerName="maas-api" Sep 15 11:52:20.994956 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:20.994868 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e6eb202-66d8-4539-9515-7d68f83fb14f" containerName="payload-pre-processing" Sep 15 11:52:20.994956 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:20.994876 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e6eb202-66d8-4539-9515-7d68f83fb14f" containerName="payload-pre-processing" Sep 15 11:52:20.994956 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:20.994890 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18cde059-c1f1-4af0-8c25-62ef4e37a9fd" containerName="payload-processing" Sep 15 11:52:20.994956 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:20.994900 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="18cde059-c1f1-4af0-8c25-62ef4e37a9fd" containerName="payload-processing" Sep 15 11:52:20.995331 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:20.994986 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e6eb202-66d8-4539-9515-7d68f83fb14f" containerName="payload-pre-processing" Sep 15 11:52:20.995331 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:20.994998 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5578f590-384f-4b86-bcf8-7b0147e5f3e7" containerName="maas-api" Sep 15 11:52:20.995331 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:20.995006 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="18cde059-c1f1-4af0-8c25-62ef4e37a9fd" containerName="payload-processing" Sep 15 11:52:20.997905 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:20.997883 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.000991 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.000973 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f6336aa2-serving-cert\"" Sep 15 11:52:21.015178 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.015150 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg"] Sep 15 11:52:21.076539 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.076506 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85066214-2ad7-42bd-8eb1-65b07368f344-openshift-service-ca\") pod \"maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.076698 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.076548 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-metrics-certs\") pod \"maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.076698 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.076581 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6w9k\" (UniqueName: \"kubernetes.io/projected/85066214-2ad7-42bd-8eb1-65b07368f344-kube-api-access-k6w9k\") pod \"maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.076698 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.076621 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-maas-api-tls\") pod \"maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.177732 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.177698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-maas-api-tls\") pod \"maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.177915 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.177761 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85066214-2ad7-42bd-8eb1-65b07368f344-openshift-service-ca\") pod \"maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.177915 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.177789 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-metrics-certs\") pod \"maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.177915 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.177819 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6w9k\" (UniqueName: \"kubernetes.io/projected/85066214-2ad7-42bd-8eb1-65b07368f344-kube-api-access-k6w9k\") pod \"maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.177915 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:52:21.177857 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-serving-cert: secret "maas-api-e2e-derive-f6336aa2-serving-cert" not found Sep 15 11:52:21.178361 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:52:21.177953 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-maas-api-tls podName:85066214-2ad7-42bd-8eb1-65b07368f344 nodeName:}" failed. No retries permitted until 2026-09-15 11:52:21.677931772 +0000 UTC m=+928.907858289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-maas-api-tls") pod "maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" (UID: "85066214-2ad7-42bd-8eb1-65b07368f344") : secret "maas-api-e2e-derive-f6336aa2-serving-cert" not found Sep 15 11:52:21.178606 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.178583 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85066214-2ad7-42bd-8eb1-65b07368f344-openshift-service-ca\") pod \"maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.180196 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.180178 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-metrics-certs\") pod \"maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.188977 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.188947 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6w9k\" (UniqueName: \"kubernetes.io/projected/85066214-2ad7-42bd-8eb1-65b07368f344-kube-api-access-k6w9k\") pod \"maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.682112 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.682079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-maas-api-tls\") pod \"maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.684691 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.684666 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-maas-api-tls\") pod \"maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.821639 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.821601 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr"] Sep 15 11:52:21.825000 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.824983 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" Sep 15 11:52:21.827454 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.827432 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f6336aa2-dockercfg-47wwp\"" Sep 15 11:52:21.827810 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.827788 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f6336aa2\"" Sep 15 11:52:21.836799 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.836775 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr"] Sep 15 11:52:21.884591 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.884556 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3549ce78-2d2f-4a3d-8057-38bf65624581-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr\" (UID: \"3549ce78-2d2f-4a3d-8057-38bf65624581\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" Sep 15 11:52:21.884765 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.884613 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spc54\" (UniqueName: \"kubernetes.io/projected/3549ce78-2d2f-4a3d-8057-38bf65624581-kube-api-access-spc54\") pod \"payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr\" (UID: \"3549ce78-2d2f-4a3d-8057-38bf65624581\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" Sep 15 11:52:21.908532 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.908503 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:21.986493 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.986459 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3549ce78-2d2f-4a3d-8057-38bf65624581-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr\" (UID: \"3549ce78-2d2f-4a3d-8057-38bf65624581\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" Sep 15 11:52:21.986621 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.986530 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spc54\" (UniqueName: \"kubernetes.io/projected/3549ce78-2d2f-4a3d-8057-38bf65624581-kube-api-access-spc54\") pod \"payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr\" (UID: \"3549ce78-2d2f-4a3d-8057-38bf65624581\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" Sep 15 11:52:21.987244 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.987224 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3549ce78-2d2f-4a3d-8057-38bf65624581-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr\" (UID: \"3549ce78-2d2f-4a3d-8057-38bf65624581\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" Sep 15 11:52:21.995335 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:21.995308 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spc54\" (UniqueName: \"kubernetes.io/projected/3549ce78-2d2f-4a3d-8057-38bf65624581-kube-api-access-spc54\") pod \"payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr\" (UID: \"3549ce78-2d2f-4a3d-8057-38bf65624581\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" Sep 15 11:52:22.136390 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:22.136356 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" Sep 15 11:52:22.192521 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:22.192349 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg"] Sep 15 11:52:22.343537 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:22.343502 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" event={"ID":"85066214-2ad7-42bd-8eb1-65b07368f344","Type":"ContainerStarted","Data":"5768a440b9f44b3ee50b4d23387f289dc8edaed251228c88aa7980a55c59019b"} Sep 15 11:52:22.416783 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:22.416752 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr"] Sep 15 11:52:22.419843 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:52:22.419816 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3549ce78_2d2f_4a3d_8057_38bf65624581.slice/crio-dd5f9712eec515653db78fd33e1d54b0c17784a04edf06cb51e291b581d19ce0 WatchSource:0}: Error finding container dd5f9712eec515653db78fd33e1d54b0c17784a04edf06cb51e291b581d19ce0: Status 404 returned error can't find the container with id dd5f9712eec515653db78fd33e1d54b0c17784a04edf06cb51e291b581d19ce0 Sep 15 11:52:23.350080 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:23.350046 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" event={"ID":"85066214-2ad7-42bd-8eb1-65b07368f344","Type":"ContainerStarted","Data":"016ae83bdf759c01951506e148e91d4af824c4985c29c9e187e85a785e307a1d"} Sep 15 11:52:23.350531 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:23.350157 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:23.351693 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:23.351671 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" event={"ID":"3549ce78-2d2f-4a3d-8057-38bf65624581","Type":"ContainerStarted","Data":"80b3ece977d6e1cf0d3449c9146072cfe77f8b51c8b647343b1cd8379156a516"} Sep 15 11:52:23.351813 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:23.351710 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" event={"ID":"3549ce78-2d2f-4a3d-8057-38bf65624581","Type":"ContainerStarted","Data":"dd5f9712eec515653db78fd33e1d54b0c17784a04edf06cb51e291b581d19ce0"} Sep 15 11:52:23.351813 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:23.351739 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" Sep 15 11:52:23.375128 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:23.375078 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" podStartSLOduration=2.970119327 podStartE2EDuration="3.375064577s" totalImagesPullingTime="404.94525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:23.372534271 +0000 UTC m=+930.602460808" watchObservedRunningTime="2026-09-15 11:52:23.375064577 +0000 UTC m=+930.604991112" Sep 15 11:52:23.394419 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:23.394376 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" podStartSLOduration=2.394363676 podStartE2EDuration="2.394363676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:23.393566372 +0000 UTC m=+930.623492908" watchObservedRunningTime="2026-09-15 11:52:23.394363676 +0000 UTC m=+930.624290270" Sep 15 11:52:29.362425 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:29.362395 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:34.359471 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:34.359435 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" Sep 15 11:52:51.358708 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.358663 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg"] Sep 15 11:52:51.359255 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.358880 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" podUID="85066214-2ad7-42bd-8eb1-65b07368f344" containerName="maas-api" containerID="cri-o://016ae83bdf759c01951506e148e91d4af824c4985c29c9e187e85a785e307a1d" gracePeriod=30 Sep 15 11:52:51.461912 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.461739 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr"] Sep 15 11:52:51.462218 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.462168 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" podUID="3549ce78-2d2f-4a3d-8057-38bf65624581" containerName="payload-pre-processing" containerID="cri-o://80b3ece977d6e1cf0d3449c9146072cfe77f8b51c8b647343b1cd8379156a516" gracePeriod=30 Sep 15 11:52:51.495817 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.495778 2573 generic.go:356] "Generic (PLEG): container finished" podID="85066214-2ad7-42bd-8eb1-65b07368f344" containerID="016ae83bdf759c01951506e148e91d4af824c4985c29c9e187e85a785e307a1d" exitCode=0 Sep 15 11:52:51.495962 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.495821 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" event={"ID":"85066214-2ad7-42bd-8eb1-65b07368f344","Type":"ContainerDied","Data":"016ae83bdf759c01951506e148e91d4af824c4985c29c9e187e85a785e307a1d"} Sep 15 11:52:51.599352 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.599319 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:51.664803 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.664771 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85066214-2ad7-42bd-8eb1-65b07368f344-openshift-service-ca\") pod \"85066214-2ad7-42bd-8eb1-65b07368f344\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " Sep 15 11:52:51.665137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.664812 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6w9k\" (UniqueName: \"kubernetes.io/projected/85066214-2ad7-42bd-8eb1-65b07368f344-kube-api-access-k6w9k\") pod \"85066214-2ad7-42bd-8eb1-65b07368f344\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " Sep 15 11:52:51.665137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.664905 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-maas-api-tls\") pod \"85066214-2ad7-42bd-8eb1-65b07368f344\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " Sep 15 11:52:51.665137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.664959 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-metrics-certs\") pod \"85066214-2ad7-42bd-8eb1-65b07368f344\" (UID: \"85066214-2ad7-42bd-8eb1-65b07368f344\") " Sep 15 11:52:51.665405 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.665372 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85066214-2ad7-42bd-8eb1-65b07368f344-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85066214-2ad7-42bd-8eb1-65b07368f344" (UID: "85066214-2ad7-42bd-8eb1-65b07368f344"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:52:51.667148 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.667116 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85066214-2ad7-42bd-8eb1-65b07368f344" (UID: "85066214-2ad7-42bd-8eb1-65b07368f344"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:52:51.667898 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.667829 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85066214-2ad7-42bd-8eb1-65b07368f344-kube-api-access-k6w9k" (OuterVolumeSpecName: "kube-api-access-k6w9k") pod "85066214-2ad7-42bd-8eb1-65b07368f344" (UID: "85066214-2ad7-42bd-8eb1-65b07368f344"). InnerVolumeSpecName "kube-api-access-k6w9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:52:51.668224 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.668196 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "85066214-2ad7-42bd-8eb1-65b07368f344" (UID: "85066214-2ad7-42bd-8eb1-65b07368f344"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:52:51.708579 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.708556 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" Sep 15 11:52:51.766238 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.766214 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spc54\" (UniqueName: \"kubernetes.io/projected/3549ce78-2d2f-4a3d-8057-38bf65624581-kube-api-access-spc54\") pod \"3549ce78-2d2f-4a3d-8057-38bf65624581\" (UID: \"3549ce78-2d2f-4a3d-8057-38bf65624581\") " Sep 15 11:52:51.766392 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.766253 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3549ce78-2d2f-4a3d-8057-38bf65624581-plugins-config-volume\") pod \"3549ce78-2d2f-4a3d-8057-38bf65624581\" (UID: \"3549ce78-2d2f-4a3d-8057-38bf65624581\") " Sep 15 11:52:51.766544 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.766522 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85066214-2ad7-42bd-8eb1-65b07368f344-openshift-service-ca\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:51.766589 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.766551 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6w9k\" (UniqueName: \"kubernetes.io/projected/85066214-2ad7-42bd-8eb1-65b07368f344-kube-api-access-k6w9k\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:51.766639 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.766568 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-maas-api-tls\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:51.766639 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.766598 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3549ce78-2d2f-4a3d-8057-38bf65624581-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3549ce78-2d2f-4a3d-8057-38bf65624581" (UID: "3549ce78-2d2f-4a3d-8057-38bf65624581"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:52:51.766639 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.766609 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85066214-2ad7-42bd-8eb1-65b07368f344-metrics-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:51.768360 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.768337 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3549ce78-2d2f-4a3d-8057-38bf65624581-kube-api-access-spc54" (OuterVolumeSpecName: "kube-api-access-spc54") pod "3549ce78-2d2f-4a3d-8057-38bf65624581" (UID: "3549ce78-2d2f-4a3d-8057-38bf65624581"). InnerVolumeSpecName "kube-api-access-spc54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:52:51.867669 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.867634 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spc54\" (UniqueName: \"kubernetes.io/projected/3549ce78-2d2f-4a3d-8057-38bf65624581-kube-api-access-spc54\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:51.867669 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:51.867668 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3549ce78-2d2f-4a3d-8057-38bf65624581-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:52:52.502035 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.501997 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" event={"ID":"85066214-2ad7-42bd-8eb1-65b07368f344","Type":"ContainerDied","Data":"5768a440b9f44b3ee50b4d23387f289dc8edaed251228c88aa7980a55c59019b"} Sep 15 11:52:52.502451 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.502005 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg" Sep 15 11:52:52.502451 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.502074 2573 scope.go:117] "RemoveContainer" containerID="016ae83bdf759c01951506e148e91d4af824c4985c29c9e187e85a785e307a1d" Sep 15 11:52:52.503524 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.503327 2573 generic.go:356] "Generic (PLEG): container finished" podID="3549ce78-2d2f-4a3d-8057-38bf65624581" containerID="80b3ece977d6e1cf0d3449c9146072cfe77f8b51c8b647343b1cd8379156a516" exitCode=0 Sep 15 11:52:52.503524 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.503373 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" Sep 15 11:52:52.503524 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.503406 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" event={"ID":"3549ce78-2d2f-4a3d-8057-38bf65624581","Type":"ContainerDied","Data":"80b3ece977d6e1cf0d3449c9146072cfe77f8b51c8b647343b1cd8379156a516"} Sep 15 11:52:52.503524 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.503453 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr" event={"ID":"3549ce78-2d2f-4a3d-8057-38bf65624581","Type":"ContainerDied","Data":"dd5f9712eec515653db78fd33e1d54b0c17784a04edf06cb51e291b581d19ce0"} Sep 15 11:52:52.512958 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.512938 2573 scope.go:117] "RemoveContainer" containerID="80b3ece977d6e1cf0d3449c9146072cfe77f8b51c8b647343b1cd8379156a516" Sep 15 11:52:52.521697 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.521674 2573 scope.go:117] "RemoveContainer" containerID="80b3ece977d6e1cf0d3449c9146072cfe77f8b51c8b647343b1cd8379156a516" Sep 15 11:52:52.521984 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:52:52.521962 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b3ece977d6e1cf0d3449c9146072cfe77f8b51c8b647343b1cd8379156a516\": container with ID starting with 80b3ece977d6e1cf0d3449c9146072cfe77f8b51c8b647343b1cd8379156a516 not found: ID does not exist" containerID="80b3ece977d6e1cf0d3449c9146072cfe77f8b51c8b647343b1cd8379156a516" Sep 15 11:52:52.522081 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.521997 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b3ece977d6e1cf0d3449c9146072cfe77f8b51c8b647343b1cd8379156a516"} err="failed to get container status \"80b3ece977d6e1cf0d3449c9146072cfe77f8b51c8b647343b1cd8379156a516\": rpc error: code = NotFound desc = could not find container \"80b3ece977d6e1cf0d3449c9146072cfe77f8b51c8b647343b1cd8379156a516\": container with ID starting with 80b3ece977d6e1cf0d3449c9146072cfe77f8b51c8b647343b1cd8379156a516 not found: ID does not exist" Sep 15 11:52:52.537301 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.537273 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr"] Sep 15 11:52:52.539759 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.539736 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f6336aa2-555489fff4-lvjxr"] Sep 15 11:52:52.555560 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.555533 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg"] Sep 15 11:52:52.561997 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:52.561976 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f6336aa2-794c8557f7-zkxlg"] Sep 15 11:52:53.235905 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:53.235870 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3549ce78-2d2f-4a3d-8057-38bf65624581" path="/var/lib/kubelet/pods/3549ce78-2d2f-4a3d-8057-38bf65624581/volumes" Sep 15 11:52:53.236318 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:52:53.236302 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85066214-2ad7-42bd-8eb1-65b07368f344" path="/var/lib/kubelet/pods/85066214-2ad7-42bd-8eb1-65b07368f344/volumes" Sep 15 11:53:04.309980 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:04.309946 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8"] Sep 15 11:53:05.297635 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:05.297604 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn"] Sep 15 11:53:19.328720 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.328607 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" podUID="87bc530c-da3b-4502-9cb8-6410dc1d0fef" containerName="main" containerID="cri-o://2dfb68232c3a02b0fd4a583c19f03afbe0301bb2e67b9a77b45dd1e57fd6d50f" gracePeriod=45 Sep 15 11:53:19.575104 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.575076 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:53:19.626167 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.626095 2573 generic.go:356] "Generic (PLEG): container finished" podID="87bc530c-da3b-4502-9cb8-6410dc1d0fef" containerID="2dfb68232c3a02b0fd4a583c19f03afbe0301bb2e67b9a77b45dd1e57fd6d50f" exitCode=0 Sep 15 11:53:19.626167 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.626133 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" Sep 15 11:53:19.626348 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.626169 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" event={"ID":"87bc530c-da3b-4502-9cb8-6410dc1d0fef","Type":"ContainerDied","Data":"2dfb68232c3a02b0fd4a583c19f03afbe0301bb2e67b9a77b45dd1e57fd6d50f"} Sep 15 11:53:19.626348 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.626213 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8" event={"ID":"87bc530c-da3b-4502-9cb8-6410dc1d0fef","Type":"ContainerDied","Data":"483b7e0350f7a2cc0e8c387c7ccd182a61bd82124b21ff6ddbd401abbfbe318b"} Sep 15 11:53:19.626348 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.626230 2573 scope.go:117] "RemoveContainer" containerID="2dfb68232c3a02b0fd4a583c19f03afbe0301bb2e67b9a77b45dd1e57fd6d50f" Sep 15 11:53:19.634110 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.634093 2573 scope.go:117] "RemoveContainer" containerID="2dfb68232c3a02b0fd4a583c19f03afbe0301bb2e67b9a77b45dd1e57fd6d50f" Sep 15 11:53:19.634362 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:53:19.634342 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dfb68232c3a02b0fd4a583c19f03afbe0301bb2e67b9a77b45dd1e57fd6d50f\": container with ID starting with 2dfb68232c3a02b0fd4a583c19f03afbe0301bb2e67b9a77b45dd1e57fd6d50f not found: ID does not exist" containerID="2dfb68232c3a02b0fd4a583c19f03afbe0301bb2e67b9a77b45dd1e57fd6d50f" Sep 15 11:53:19.634426 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.634374 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dfb68232c3a02b0fd4a583c19f03afbe0301bb2e67b9a77b45dd1e57fd6d50f"} err="failed to get container status \"2dfb68232c3a02b0fd4a583c19f03afbe0301bb2e67b9a77b45dd1e57fd6d50f\": rpc error: code = NotFound desc = could not find container \"2dfb68232c3a02b0fd4a583c19f03afbe0301bb2e67b9a77b45dd1e57fd6d50f\": container with ID starting with 2dfb68232c3a02b0fd4a583c19f03afbe0301bb2e67b9a77b45dd1e57fd6d50f not found: ID does not exist" Sep 15 11:53:19.702439 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.702407 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-dshm\") pod \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " Sep 15 11:53:19.702439 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.702444 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87bc530c-da3b-4502-9cb8-6410dc1d0fef-tls-certs\") pod \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " Sep 15 11:53:19.702662 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.702488 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phgbh\" (UniqueName: \"kubernetes.io/projected/87bc530c-da3b-4502-9cb8-6410dc1d0fef-kube-api-access-phgbh\") pod \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " Sep 15 11:53:19.702662 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.702506 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-model-cache\") pod \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " Sep 15 11:53:19.702662 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.702531 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-tmp-dir\") pod \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " Sep 15 11:53:19.702662 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.702566 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-home\") pod \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\" (UID: \"87bc530c-da3b-4502-9cb8-6410dc1d0fef\") " Sep 15 11:53:19.702875 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.702830 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-model-cache" (OuterVolumeSpecName: "model-cache") pod "87bc530c-da3b-4502-9cb8-6410dc1d0fef" (UID: "87bc530c-da3b-4502-9cb8-6410dc1d0fef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:53:19.702875 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.702842 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87bc530c-da3b-4502-9cb8-6410dc1d0fef" (UID: "87bc530c-da3b-4502-9cb8-6410dc1d0fef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:53:19.702994 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.702932 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-home" (OuterVolumeSpecName: "home") pod "87bc530c-da3b-4502-9cb8-6410dc1d0fef" (UID: "87bc530c-da3b-4502-9cb8-6410dc1d0fef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:53:19.704751 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.704731 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-dshm" (OuterVolumeSpecName: "dshm") pod "87bc530c-da3b-4502-9cb8-6410dc1d0fef" (UID: "87bc530c-da3b-4502-9cb8-6410dc1d0fef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:53:19.704835 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.704769 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87bc530c-da3b-4502-9cb8-6410dc1d0fef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87bc530c-da3b-4502-9cb8-6410dc1d0fef" (UID: "87bc530c-da3b-4502-9cb8-6410dc1d0fef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:53:19.704876 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.704848 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87bc530c-da3b-4502-9cb8-6410dc1d0fef-kube-api-access-phgbh" (OuterVolumeSpecName: "kube-api-access-phgbh") pod "87bc530c-da3b-4502-9cb8-6410dc1d0fef" (UID: "87bc530c-da3b-4502-9cb8-6410dc1d0fef"). InnerVolumeSpecName "kube-api-access-phgbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:53:19.803425 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.803393 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phgbh\" (UniqueName: \"kubernetes.io/projected/87bc530c-da3b-4502-9cb8-6410dc1d0fef-kube-api-access-phgbh\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:19.803425 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.803422 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-model-cache\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:19.803425 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.803433 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-tmp-dir\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:19.803648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.803440 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-home\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:19.803648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.803448 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87bc530c-da3b-4502-9cb8-6410dc1d0fef-dshm\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:19.803648 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.803455 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87bc530c-da3b-4502-9cb8-6410dc1d0fef-tls-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:19.953741 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.953704 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8"] Sep 15 11:53:19.960982 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:19.960956 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-155ffe/test-model-155ffe-kserve-6d6bc78465-ghwf8"] Sep 15 11:53:20.316184 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.316144 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" podUID="7ace007a-41eb-4344-af2f-3cd62646196a" containerName="main" containerID="cri-o://265c0015245886566367680f888e9ebcd3987e4a93dbded496b13629ecc0240e" gracePeriod=45 Sep 15 11:53:20.561760 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.561735 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:53:20.632724 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.632649 2573 generic.go:356] "Generic (PLEG): container finished" podID="7ace007a-41eb-4344-af2f-3cd62646196a" containerID="265c0015245886566367680f888e9ebcd3987e4a93dbded496b13629ecc0240e" exitCode=0 Sep 15 11:53:20.632724 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.632698 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" Sep 15 11:53:20.632903 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.632733 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" event={"ID":"7ace007a-41eb-4344-af2f-3cd62646196a","Type":"ContainerDied","Data":"265c0015245886566367680f888e9ebcd3987e4a93dbded496b13629ecc0240e"} Sep 15 11:53:20.632903 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.632780 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn" event={"ID":"7ace007a-41eb-4344-af2f-3cd62646196a","Type":"ContainerDied","Data":"64257b92b9236f4f33a37acbbdda49dc6dde2f54f023cd018d2ba4b9412437b2"} Sep 15 11:53:20.632903 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.632798 2573 scope.go:117] "RemoveContainer" containerID="265c0015245886566367680f888e9ebcd3987e4a93dbded496b13629ecc0240e" Sep 15 11:53:20.643386 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.643364 2573 scope.go:117] "RemoveContainer" containerID="265c0015245886566367680f888e9ebcd3987e4a93dbded496b13629ecc0240e" Sep 15 11:53:20.643655 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:53:20.643636 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265c0015245886566367680f888e9ebcd3987e4a93dbded496b13629ecc0240e\": container with ID starting with 265c0015245886566367680f888e9ebcd3987e4a93dbded496b13629ecc0240e not found: ID does not exist" containerID="265c0015245886566367680f888e9ebcd3987e4a93dbded496b13629ecc0240e" Sep 15 11:53:20.643710 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.643663 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265c0015245886566367680f888e9ebcd3987e4a93dbded496b13629ecc0240e"} err="failed to get container status \"265c0015245886566367680f888e9ebcd3987e4a93dbded496b13629ecc0240e\": rpc error: code = NotFound desc = could not find container \"265c0015245886566367680f888e9ebcd3987e4a93dbded496b13629ecc0240e\": container with ID starting with 265c0015245886566367680f888e9ebcd3987e4a93dbded496b13629ecc0240e not found: ID does not exist" Sep 15 11:53:20.710617 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.710590 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-home\") pod \"7ace007a-41eb-4344-af2f-3cd62646196a\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " Sep 15 11:53:20.710767 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.710621 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ace007a-41eb-4344-af2f-3cd62646196a-tls-certs\") pod \"7ace007a-41eb-4344-af2f-3cd62646196a\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " Sep 15 11:53:20.710767 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.710659 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-dshm\") pod \"7ace007a-41eb-4344-af2f-3cd62646196a\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " Sep 15 11:53:20.710767 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.710676 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-model-cache\") pod \"7ace007a-41eb-4344-af2f-3cd62646196a\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " Sep 15 11:53:20.710767 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.710692 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-tmp-dir\") pod \"7ace007a-41eb-4344-af2f-3cd62646196a\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " Sep 15 11:53:20.710767 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.710725 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzhkq\" (UniqueName: \"kubernetes.io/projected/7ace007a-41eb-4344-af2f-3cd62646196a-kube-api-access-zzhkq\") pod \"7ace007a-41eb-4344-af2f-3cd62646196a\" (UID: \"7ace007a-41eb-4344-af2f-3cd62646196a\") " Sep 15 11:53:20.711055 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.710947 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-home" (OuterVolumeSpecName: "home") pod "7ace007a-41eb-4344-af2f-3cd62646196a" (UID: "7ace007a-41eb-4344-af2f-3cd62646196a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:53:20.711055 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.710961 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ace007a-41eb-4344-af2f-3cd62646196a" (UID: "7ace007a-41eb-4344-af2f-3cd62646196a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:53:20.711055 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.711043 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ace007a-41eb-4344-af2f-3cd62646196a" (UID: "7ace007a-41eb-4344-af2f-3cd62646196a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:53:20.712806 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.712778 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-dshm" (OuterVolumeSpecName: "dshm") pod "7ace007a-41eb-4344-af2f-3cd62646196a" (UID: "7ace007a-41eb-4344-af2f-3cd62646196a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:53:20.712867 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.712840 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ace007a-41eb-4344-af2f-3cd62646196a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ace007a-41eb-4344-af2f-3cd62646196a" (UID: "7ace007a-41eb-4344-af2f-3cd62646196a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:53:20.712906 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.712879 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ace007a-41eb-4344-af2f-3cd62646196a-kube-api-access-zzhkq" (OuterVolumeSpecName: "kube-api-access-zzhkq") pod "7ace007a-41eb-4344-af2f-3cd62646196a" (UID: "7ace007a-41eb-4344-af2f-3cd62646196a"). InnerVolumeSpecName "kube-api-access-zzhkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:53:20.812397 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.812366 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-home\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:20.812397 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.812395 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ace007a-41eb-4344-af2f-3cd62646196a-tls-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:20.812548 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.812406 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-dshm\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:20.812548 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.812416 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-model-cache\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:20.812548 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.812423 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ace007a-41eb-4344-af2f-3cd62646196a-tmp-dir\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:20.812548 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.812432 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzhkq\" (UniqueName: \"kubernetes.io/projected/7ace007a-41eb-4344-af2f-3cd62646196a-kube-api-access-zzhkq\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:20.963662 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.963630 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn"] Sep 15 11:53:20.971416 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:20.971392 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1b147a/test-model-1b147a-kserve-7c99979865-rjcgn"] Sep 15 11:53:21.233747 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:21.233670 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ace007a-41eb-4344-af2f-3cd62646196a" path="/var/lib/kubelet/pods/7ace007a-41eb-4344-af2f-3cd62646196a/volumes" Sep 15 11:53:21.234075 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:21.234063 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87bc530c-da3b-4502-9cb8-6410dc1d0fef" path="/var/lib/kubelet/pods/87bc530c-da3b-4502-9cb8-6410dc1d0fef/volumes" Sep 15 11:53:34.885722 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:34.885689 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn"] Sep 15 11:53:34.886085 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:34.885917 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" podUID="6cc8a54d-60ec-4d88-aa20-1da7b375d912" containerName="maas-api" containerID="cri-o://96d990507f050f2e65c68402ca808ec7040e24fc9c60474551959a062a63b274" gracePeriod=30 Sep 15 11:53:35.137209 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.137151 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:53:35.232370 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.232341 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc8a54d-60ec-4d88-aa20-1da7b375d912-metrics-certs\") pod \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " Sep 15 11:53:35.232560 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.232438 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc8a54d-60ec-4d88-aa20-1da7b375d912-openshift-service-ca\") pod \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " Sep 15 11:53:35.232560 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.232474 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-262xj\" (UniqueName: \"kubernetes.io/projected/6cc8a54d-60ec-4d88-aa20-1da7b375d912-kube-api-access-262xj\") pod \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " Sep 15 11:53:35.232560 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.232505 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc8a54d-60ec-4d88-aa20-1da7b375d912-maas-api-tls\") pod \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\" (UID: \"6cc8a54d-60ec-4d88-aa20-1da7b375d912\") " Sep 15 11:53:35.232840 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.232800 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc8a54d-60ec-4d88-aa20-1da7b375d912-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6cc8a54d-60ec-4d88-aa20-1da7b375d912" (UID: "6cc8a54d-60ec-4d88-aa20-1da7b375d912"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:53:35.234635 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.234597 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc8a54d-60ec-4d88-aa20-1da7b375d912-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6cc8a54d-60ec-4d88-aa20-1da7b375d912" (UID: "6cc8a54d-60ec-4d88-aa20-1da7b375d912"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:53:35.234762 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.234658 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc8a54d-60ec-4d88-aa20-1da7b375d912-kube-api-access-262xj" (OuterVolumeSpecName: "kube-api-access-262xj") pod "6cc8a54d-60ec-4d88-aa20-1da7b375d912" (UID: "6cc8a54d-60ec-4d88-aa20-1da7b375d912"). InnerVolumeSpecName "kube-api-access-262xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:53:35.234950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.234930 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc8a54d-60ec-4d88-aa20-1da7b375d912-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6cc8a54d-60ec-4d88-aa20-1da7b375d912" (UID: "6cc8a54d-60ec-4d88-aa20-1da7b375d912"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:53:35.333355 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.333310 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc8a54d-60ec-4d88-aa20-1da7b375d912-openshift-service-ca\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:35.333355 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.333345 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-262xj\" (UniqueName: \"kubernetes.io/projected/6cc8a54d-60ec-4d88-aa20-1da7b375d912-kube-api-access-262xj\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:35.333355 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.333359 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc8a54d-60ec-4d88-aa20-1da7b375d912-maas-api-tls\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:35.333625 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.333371 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc8a54d-60ec-4d88-aa20-1da7b375d912-metrics-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:53:35.700738 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.700707 2573 generic.go:356] "Generic (PLEG): container finished" podID="6cc8a54d-60ec-4d88-aa20-1da7b375d912" containerID="96d990507f050f2e65c68402ca808ec7040e24fc9c60474551959a062a63b274" exitCode=0 Sep 15 11:53:35.700886 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.700750 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" Sep 15 11:53:35.700886 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.700794 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" event={"ID":"6cc8a54d-60ec-4d88-aa20-1da7b375d912","Type":"ContainerDied","Data":"96d990507f050f2e65c68402ca808ec7040e24fc9c60474551959a062a63b274"} Sep 15 11:53:35.700886 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.700868 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn" event={"ID":"6cc8a54d-60ec-4d88-aa20-1da7b375d912","Type":"ContainerDied","Data":"baecaeb4e01464ab701a00c239c3eb456eca912a0dd32bc39f24634d1b8c3307"} Sep 15 11:53:35.701063 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.700894 2573 scope.go:117] "RemoveContainer" containerID="96d990507f050f2e65c68402ca808ec7040e24fc9c60474551959a062a63b274" Sep 15 11:53:35.710904 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.710887 2573 scope.go:117] "RemoveContainer" containerID="96d990507f050f2e65c68402ca808ec7040e24fc9c60474551959a062a63b274" Sep 15 11:53:35.711207 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:53:35.711189 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d990507f050f2e65c68402ca808ec7040e24fc9c60474551959a062a63b274\": container with ID starting with 96d990507f050f2e65c68402ca808ec7040e24fc9c60474551959a062a63b274 not found: ID does not exist" containerID="96d990507f050f2e65c68402ca808ec7040e24fc9c60474551959a062a63b274" Sep 15 11:53:35.711259 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.711215 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d990507f050f2e65c68402ca808ec7040e24fc9c60474551959a062a63b274"} err="failed to get container status \"96d990507f050f2e65c68402ca808ec7040e24fc9c60474551959a062a63b274\": rpc error: code = NotFound desc = could not find container \"96d990507f050f2e65c68402ca808ec7040e24fc9c60474551959a062a63b274\": container with ID starting with 96d990507f050f2e65c68402ca808ec7040e24fc9c60474551959a062a63b274 not found: ID does not exist" Sep 15 11:53:35.738859 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.738830 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn"] Sep 15 11:53:35.747113 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:35.747074 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1b147a-57d5bbf6bb-phrfn"] Sep 15 11:53:37.234633 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:37.234597 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cc8a54d-60ec-4d88-aa20-1da7b375d912" path="/var/lib/kubelet/pods/6cc8a54d-60ec-4d88-aa20-1da7b375d912/volumes" Sep 15 11:53:48.350921 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.350885 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4"] Sep 15 11:53:48.351398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351252 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87bc530c-da3b-4502-9cb8-6410dc1d0fef" containerName="main" Sep 15 11:53:48.351398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351267 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="87bc530c-da3b-4502-9cb8-6410dc1d0fef" containerName="main" Sep 15 11:53:48.351398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351288 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ace007a-41eb-4344-af2f-3cd62646196a" containerName="main" Sep 15 11:53:48.351398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351295 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ace007a-41eb-4344-af2f-3cd62646196a" containerName="main" Sep 15 11:53:48.351398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351302 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc8a54d-60ec-4d88-aa20-1da7b375d912" containerName="maas-api" Sep 15 11:53:48.351398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351308 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc8a54d-60ec-4d88-aa20-1da7b375d912" containerName="maas-api" Sep 15 11:53:48.351398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351315 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3549ce78-2d2f-4a3d-8057-38bf65624581" containerName="payload-pre-processing" Sep 15 11:53:48.351398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351320 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3549ce78-2d2f-4a3d-8057-38bf65624581" containerName="payload-pre-processing" Sep 15 11:53:48.351398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351326 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85066214-2ad7-42bd-8eb1-65b07368f344" containerName="maas-api" Sep 15 11:53:48.351398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351332 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="85066214-2ad7-42bd-8eb1-65b07368f344" containerName="maas-api" Sep 15 11:53:48.351398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351385 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="85066214-2ad7-42bd-8eb1-65b07368f344" containerName="maas-api" Sep 15 11:53:48.351398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351395 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc8a54d-60ec-4d88-aa20-1da7b375d912" containerName="maas-api" Sep 15 11:53:48.351398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351401 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ace007a-41eb-4344-af2f-3cd62646196a" containerName="main" Sep 15 11:53:48.351789 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351408 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="87bc530c-da3b-4502-9cb8-6410dc1d0fef" containerName="main" Sep 15 11:53:48.351789 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.351415 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3549ce78-2d2f-4a3d-8057-38bf65624581" containerName="payload-pre-processing" Sep 15 11:53:48.355852 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.355832 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" Sep 15 11:53:48.358473 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.358453 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-42b26e-dockercfg-n6j5n\"" Sep 15 11:53:48.358605 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.358459 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-42b26e\"" Sep 15 11:53:48.373531 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.373503 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4"] Sep 15 11:53:48.387367 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.387342 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd"] Sep 15 11:53:48.390424 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.390406 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" Sep 15 11:53:48.401190 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.401168 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd"] Sep 15 11:53:48.453411 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.453377 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54zd5\" (UniqueName: \"kubernetes.io/projected/9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132-kube-api-access-54zd5\") pod \"payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd\" (UID: \"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" Sep 15 11:53:48.453582 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.453421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd\" (UID: \"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" Sep 15 11:53:48.453582 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.453443 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61f0d11-698f-4f14-b46c-2a43e73e571b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4\" (UID: \"a61f0d11-698f-4f14-b46c-2a43e73e571b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" Sep 15 11:53:48.453582 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.453469 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nq8z\" (UniqueName: \"kubernetes.io/projected/a61f0d11-698f-4f14-b46c-2a43e73e571b-kube-api-access-6nq8z\") pod \"payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4\" (UID: \"a61f0d11-698f-4f14-b46c-2a43e73e571b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" Sep 15 11:53:48.554751 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.554712 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54zd5\" (UniqueName: \"kubernetes.io/projected/9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132-kube-api-access-54zd5\") pod \"payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd\" (UID: \"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" Sep 15 11:53:48.554934 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.554777 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd\" (UID: \"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" Sep 15 11:53:48.554934 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.554808 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61f0d11-698f-4f14-b46c-2a43e73e571b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4\" (UID: \"a61f0d11-698f-4f14-b46c-2a43e73e571b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" Sep 15 11:53:48.554934 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.554836 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nq8z\" (UniqueName: \"kubernetes.io/projected/a61f0d11-698f-4f14-b46c-2a43e73e571b-kube-api-access-6nq8z\") pod \"payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4\" (UID: \"a61f0d11-698f-4f14-b46c-2a43e73e571b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" Sep 15 11:53:48.555516 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.555495 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd\" (UID: \"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" Sep 15 11:53:48.555582 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.555530 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61f0d11-698f-4f14-b46c-2a43e73e571b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4\" (UID: \"a61f0d11-698f-4f14-b46c-2a43e73e571b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" Sep 15 11:53:48.563120 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.563088 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nq8z\" (UniqueName: \"kubernetes.io/projected/a61f0d11-698f-4f14-b46c-2a43e73e571b-kube-api-access-6nq8z\") pod \"payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4\" (UID: \"a61f0d11-698f-4f14-b46c-2a43e73e571b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" Sep 15 11:53:48.563228 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.563088 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54zd5\" (UniqueName: \"kubernetes.io/projected/9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132-kube-api-access-54zd5\") pod \"payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd\" (UID: \"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" Sep 15 11:53:48.665716 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.665632 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" Sep 15 11:53:48.700888 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.700855 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" Sep 15 11:53:48.936890 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.936864 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4"] Sep 15 11:53:48.938950 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:53:48.938913 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61f0d11_698f_4f14_b46c_2a43e73e571b.slice/crio-deeadf4772746e32efd43736d98033712e0155218d7f2b84eebde25a230fd492 WatchSource:0}: Error finding container deeadf4772746e32efd43736d98033712e0155218d7f2b84eebde25a230fd492: Status 404 returned error can't find the container with id deeadf4772746e32efd43736d98033712e0155218d7f2b84eebde25a230fd492 Sep 15 11:53:48.979195 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:48.979168 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd"] Sep 15 11:53:48.981981 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:53:48.981958 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1200c8_efbb_4caf_ae6f_a7fd3ee2d132.slice/crio-34852df419415b1d1b4d23fe9914624a4fdd826ab62a1d8d087e9eb9122bdb2a WatchSource:0}: Error finding container 34852df419415b1d1b4d23fe9914624a4fdd826ab62a1d8d087e9eb9122bdb2a: Status 404 returned error can't find the container with id 34852df419415b1d1b4d23fe9914624a4fdd826ab62a1d8d087e9eb9122bdb2a Sep 15 11:53:49.766872 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:49.766836 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" event={"ID":"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132","Type":"ContainerStarted","Data":"1a65c18aefc82b479429d770bf35f391cd2ae86d45ca41dd4b39bb1e51157fb7"} Sep 15 11:53:49.767341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:49.766884 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" event={"ID":"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132","Type":"ContainerStarted","Data":"34852df419415b1d1b4d23fe9914624a4fdd826ab62a1d8d087e9eb9122bdb2a"} Sep 15 11:53:49.767341 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:49.766935 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" Sep 15 11:53:49.768610 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:49.768583 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" event={"ID":"a61f0d11-698f-4f14-b46c-2a43e73e571b","Type":"ContainerStarted","Data":"94ed4e8f9e666bf09b797dd3492fc214d0388e215a29bebb1bcb27ea6386692d"} Sep 15 11:53:49.768725 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:49.768632 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" Sep 15 11:53:49.768725 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:49.768650 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" event={"ID":"a61f0d11-698f-4f14-b46c-2a43e73e571b","Type":"ContainerStarted","Data":"deeadf4772746e32efd43736d98033712e0155218d7f2b84eebde25a230fd492"} Sep 15 11:53:49.784166 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:49.784128 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" podStartSLOduration=1.784115303 podStartE2EDuration="1.784115303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:49.782643824 +0000 UTC m=+1017.012570361" watchObservedRunningTime="2026-09-15 11:53:49.784115303 +0000 UTC m=+1017.014041838" Sep 15 11:53:49.798219 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:53:49.798176 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" podStartSLOduration=1.798164635 podStartE2EDuration="1.798164635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:49.797259182 +0000 UTC m=+1017.027185718" watchObservedRunningTime="2026-09-15 11:53:49.798164635 +0000 UTC m=+1017.028091171" Sep 15 11:54:00.775791 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:00.775756 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" Sep 15 11:54:00.776230 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:00.775847 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" Sep 15 11:54:16.824393 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:16.824362 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5"] Sep 15 11:54:16.828916 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:16.828896 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" Sep 15 11:54:16.831189 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:16.831166 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-7b7b3c-dockercfg-txdvb\"" Sep 15 11:54:16.831189 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:16.831186 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7b7b3c\"" Sep 15 11:54:16.839168 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:16.839138 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5"] Sep 15 11:54:16.890276 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:16.890248 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a48997-c5e1-4ac9-8664-465c0373e9f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5\" (UID: \"99a48997-c5e1-4ac9-8664-465c0373e9f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" Sep 15 11:54:16.890418 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:16.890307 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6rh5\" (UniqueName: \"kubernetes.io/projected/99a48997-c5e1-4ac9-8664-465c0373e9f1-kube-api-access-c6rh5\") pod \"payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5\" (UID: \"99a48997-c5e1-4ac9-8664-465c0373e9f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" Sep 15 11:54:16.991336 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:16.991294 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rh5\" (UniqueName: \"kubernetes.io/projected/99a48997-c5e1-4ac9-8664-465c0373e9f1-kube-api-access-c6rh5\") pod \"payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5\" (UID: \"99a48997-c5e1-4ac9-8664-465c0373e9f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" Sep 15 11:54:16.991522 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:16.991388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a48997-c5e1-4ac9-8664-465c0373e9f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5\" (UID: \"99a48997-c5e1-4ac9-8664-465c0373e9f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" Sep 15 11:54:16.992274 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:16.992250 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a48997-c5e1-4ac9-8664-465c0373e9f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5\" (UID: \"99a48997-c5e1-4ac9-8664-465c0373e9f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" Sep 15 11:54:17.002157 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:17.002135 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6rh5\" (UniqueName: \"kubernetes.io/projected/99a48997-c5e1-4ac9-8664-465c0373e9f1-kube-api-access-c6rh5\") pod \"payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5\" (UID: \"99a48997-c5e1-4ac9-8664-465c0373e9f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" Sep 15 11:54:17.139478 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:17.139390 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" Sep 15 11:54:17.416884 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:17.416860 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5"] Sep 15 11:54:17.418997 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:54:17.418963 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a48997_c5e1_4ac9_8664_465c0373e9f1.slice/crio-bb5d9f06f9a668a1bfbbb63556766f89013ea3cec2c635facc9b77392774f0d3 WatchSource:0}: Error finding container bb5d9f06f9a668a1bfbbb63556766f89013ea3cec2c635facc9b77392774f0d3: Status 404 returned error can't find the container with id bb5d9f06f9a668a1bfbbb63556766f89013ea3cec2c635facc9b77392774f0d3 Sep 15 11:54:17.896686 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:17.896647 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" event={"ID":"99a48997-c5e1-4ac9-8664-465c0373e9f1","Type":"ContainerStarted","Data":"5657f869e396b7548d787c27ffad4882abeb6c8f5507b38b25bfb185f73269dc"} Sep 15 11:54:17.897127 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:17.896702 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" event={"ID":"99a48997-c5e1-4ac9-8664-465c0373e9f1","Type":"ContainerStarted","Data":"bb5d9f06f9a668a1bfbbb63556766f89013ea3cec2c635facc9b77392774f0d3"} Sep 15 11:54:17.897127 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:17.896832 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" Sep 15 11:54:17.913506 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:17.913455 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" podStartSLOduration=1.913439008 podStartE2EDuration="1.913439008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:54:17.913297789 +0000 UTC m=+1045.143224325" watchObservedRunningTime="2026-09-15 11:54:17.913439008 +0000 UTC m=+1045.143365544" Sep 15 11:54:28.903011 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:28.902978 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" Sep 15 11:54:38.280749 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.280713 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl"] Sep 15 11:54:38.284263 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.284245 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" Sep 15 11:54:38.286363 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.286339 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9064d2be\"" Sep 15 11:54:38.286967 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.286948 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9064d2be-dockercfg-j2bpw\"" Sep 15 11:54:38.294332 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.294311 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl"] Sep 15 11:54:38.311904 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.311880 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88"] Sep 15 11:54:38.315031 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.315000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" Sep 15 11:54:38.326435 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.326412 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88"] Sep 15 11:54:38.374718 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.374678 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6a02d4-4734-4f6b-a441-cec308808fce-plugins-config-volume\") pod \"payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88\" (UID: \"ab6a02d4-4734-4f6b-a441-cec308808fce\") " pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" Sep 15 11:54:38.374869 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.374726 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tgcp\" (UniqueName: \"kubernetes.io/projected/ab6a02d4-4734-4f6b-a441-cec308808fce-kube-api-access-5tgcp\") pod \"payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88\" (UID: \"ab6a02d4-4734-4f6b-a441-cec308808fce\") " pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" Sep 15 11:54:38.374869 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.374781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064fdeeb-c0ba-4c25-9246-052b10f9f3fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl\" (UID: \"064fdeeb-c0ba-4c25-9246-052b10f9f3fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" Sep 15 11:54:38.374869 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.374842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvczn\" (UniqueName: \"kubernetes.io/projected/064fdeeb-c0ba-4c25-9246-052b10f9f3fc-kube-api-access-lvczn\") pod \"payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl\" (UID: \"064fdeeb-c0ba-4c25-9246-052b10f9f3fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" Sep 15 11:54:38.476296 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.476262 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvczn\" (UniqueName: \"kubernetes.io/projected/064fdeeb-c0ba-4c25-9246-052b10f9f3fc-kube-api-access-lvczn\") pod \"payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl\" (UID: \"064fdeeb-c0ba-4c25-9246-052b10f9f3fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" Sep 15 11:54:38.476452 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.476320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6a02d4-4734-4f6b-a441-cec308808fce-plugins-config-volume\") pod \"payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88\" (UID: \"ab6a02d4-4734-4f6b-a441-cec308808fce\") " pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" Sep 15 11:54:38.476452 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.476369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tgcp\" (UniqueName: \"kubernetes.io/projected/ab6a02d4-4734-4f6b-a441-cec308808fce-kube-api-access-5tgcp\") pod \"payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88\" (UID: \"ab6a02d4-4734-4f6b-a441-cec308808fce\") " pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" Sep 15 11:54:38.476452 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.476397 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064fdeeb-c0ba-4c25-9246-052b10f9f3fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl\" (UID: \"064fdeeb-c0ba-4c25-9246-052b10f9f3fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" Sep 15 11:54:38.477334 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.477310 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064fdeeb-c0ba-4c25-9246-052b10f9f3fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl\" (UID: \"064fdeeb-c0ba-4c25-9246-052b10f9f3fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" Sep 15 11:54:38.477393 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.477310 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6a02d4-4734-4f6b-a441-cec308808fce-plugins-config-volume\") pod \"payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88\" (UID: \"ab6a02d4-4734-4f6b-a441-cec308808fce\") " pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" Sep 15 11:54:38.484694 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.484674 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvczn\" (UniqueName: \"kubernetes.io/projected/064fdeeb-c0ba-4c25-9246-052b10f9f3fc-kube-api-access-lvczn\") pod \"payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl\" (UID: \"064fdeeb-c0ba-4c25-9246-052b10f9f3fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" Sep 15 11:54:38.484879 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.484857 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tgcp\" (UniqueName: \"kubernetes.io/projected/ab6a02d4-4734-4f6b-a441-cec308808fce-kube-api-access-5tgcp\") pod \"payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88\" (UID: \"ab6a02d4-4734-4f6b-a441-cec308808fce\") " pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" Sep 15 11:54:38.594754 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.594646 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" Sep 15 11:54:38.625645 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.625620 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" Sep 15 11:54:38.866229 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.866205 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl"] Sep 15 11:54:38.867734 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:54:38.867707 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064fdeeb_c0ba_4c25_9246_052b10f9f3fc.slice/crio-f92302fb1e094015cca060c241d8d52820244883a2c81e3c53cb3afd8d8bf3d3 WatchSource:0}: Error finding container f92302fb1e094015cca060c241d8d52820244883a2c81e3c53cb3afd8d8bf3d3: Status 404 returned error can't find the container with id f92302fb1e094015cca060c241d8d52820244883a2c81e3c53cb3afd8d8bf3d3 Sep 15 11:54:38.904383 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.904363 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88"] Sep 15 11:54:38.907079 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:54:38.907052 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6a02d4_4734_4f6b_a441_cec308808fce.slice/crio-97fac44e659ffcee3e56876f628b275f6ca08474e822d04c3c7e12499fab78e0 WatchSource:0}: Error finding container 97fac44e659ffcee3e56876f628b275f6ca08474e822d04c3c7e12499fab78e0: Status 404 returned error can't find the container with id 97fac44e659ffcee3e56876f628b275f6ca08474e822d04c3c7e12499fab78e0 Sep 15 11:54:38.995290 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.995254 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" event={"ID":"ab6a02d4-4734-4f6b-a441-cec308808fce","Type":"ContainerStarted","Data":"449353d83457ec4d513b1ac0e4d5baef276ebdf4f42198e81a38dc3af7769c74"} Sep 15 11:54:38.995481 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.995319 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" event={"ID":"ab6a02d4-4734-4f6b-a441-cec308808fce","Type":"ContainerStarted","Data":"97fac44e659ffcee3e56876f628b275f6ca08474e822d04c3c7e12499fab78e0"} Sep 15 11:54:38.995481 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.995453 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" Sep 15 11:54:38.997054 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.997015 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" event={"ID":"064fdeeb-c0ba-4c25-9246-052b10f9f3fc","Type":"ContainerStarted","Data":"e3b22f1b067519d86e8200bcf0a2b466798db287fbcfe6763755fabb02874c47"} Sep 15 11:54:38.997157 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.997085 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" Sep 15 11:54:38.997157 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:38.997113 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" event={"ID":"064fdeeb-c0ba-4c25-9246-052b10f9f3fc","Type":"ContainerStarted","Data":"f92302fb1e094015cca060c241d8d52820244883a2c81e3c53cb3afd8d8bf3d3"} Sep 15 11:54:39.013661 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:39.013602 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" podStartSLOduration=1.013580716 podStartE2EDuration="1.013580716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:54:39.013064799 +0000 UTC m=+1066.242991328" watchObservedRunningTime="2026-09-15 11:54:39.013580716 +0000 UTC m=+1066.243507233" Sep 15 11:54:39.033757 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:39.033696 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" podStartSLOduration=1.033668015 podStartE2EDuration="1.033668015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:54:39.031460827 +0000 UTC m=+1066.261387374" watchObservedRunningTime="2026-09-15 11:54:39.033668015 +0000 UTC m=+1066.263594554" Sep 15 11:54:50.003818 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:50.003740 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" Sep 15 11:54:50.003818 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:50.003813 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" Sep 15 11:54:53.364238 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.364206 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd"] Sep 15 11:54:53.367305 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.367283 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.370769 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.370743 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-42b26e\"/\"auth-test-model-42b26e-kserve-self-signed-certs\"" Sep 15 11:54:53.370895 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.370742 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-42b26e\"/\"openshift-service-ca.crt\"" Sep 15 11:54:53.370895 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.370742 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-42b26e\"/\"default-dockercfg-77nkl\"" Sep 15 11:54:53.370895 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.370800 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-42b26e\"/\"kube-root-ca.crt\"" Sep 15 11:54:53.377938 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.377913 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd"] Sep 15 11:54:53.508345 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.508312 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-model-cache\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.508345 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.508341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-tmp-dir\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.508534 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.508434 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxwh9\" (UniqueName: \"kubernetes.io/projected/b19f030d-447c-4856-abb2-835a861aaa15-kube-api-access-gxwh9\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.508534 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.508478 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-dshm\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.508534 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.508495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19f030d-447c-4856-abb2-835a861aaa15-tls-certs\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.508534 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.508533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-home\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.608953 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.608921 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-dshm\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.608953 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.608954 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19f030d-447c-4856-abb2-835a861aaa15-tls-certs\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.609158 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.608992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-home\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.609158 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.609085 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-model-cache\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.609158 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.609111 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-tmp-dir\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.609294 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.609225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxwh9\" (UniqueName: \"kubernetes.io/projected/b19f030d-447c-4856-abb2-835a861aaa15-kube-api-access-gxwh9\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.609505 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.609484 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-home\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.609592 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.609506 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-tmp-dir\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.609592 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.609558 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-model-cache\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.611244 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.611225 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-dshm\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.611525 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.611505 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19f030d-447c-4856-abb2-835a861aaa15-tls-certs\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.617065 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.617000 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxwh9\" (UniqueName: \"kubernetes.io/projected/b19f030d-447c-4856-abb2-835a861aaa15-kube-api-access-gxwh9\") pod \"auth-test-model-42b26e-kserve-75c65d475-nlxgd\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.677170 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.677137 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:54:53.946486 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:53.946461 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd"] Sep 15 11:54:53.948581 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:54:53.948552 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb19f030d_447c_4856_abb2_835a861aaa15.slice/crio-ab45ec2f1f249b865b44812085e82e4a6d274868ed3b94ac6d1634a1952bc7f0 WatchSource:0}: Error finding container ab45ec2f1f249b865b44812085e82e4a6d274868ed3b94ac6d1634a1952bc7f0: Status 404 returned error can't find the container with id ab45ec2f1f249b865b44812085e82e4a6d274868ed3b94ac6d1634a1952bc7f0 Sep 15 11:54:54.066098 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:54.066068 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" event={"ID":"b19f030d-447c-4856-abb2-835a861aaa15","Type":"ContainerStarted","Data":"50294947b712265af0dbef4efc218add2c28cf8aaaa9c50c02a16f40137ab518"} Sep 15 11:54:54.066252 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:54.066109 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" event={"ID":"b19f030d-447c-4856-abb2-835a861aaa15","Type":"ContainerStarted","Data":"ab45ec2f1f249b865b44812085e82e4a6d274868ed3b94ac6d1634a1952bc7f0"} Sep 15 11:54:54.084063 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:54:54.084002 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" podStartSLOduration=1.08398606 podStartE2EDuration="1.08398606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:54:54.082543533 +0000 UTC m=+1081.312470082" watchObservedRunningTime="2026-09-15 11:54:54.08398606 +0000 UTC m=+1081.313912595" Sep 15 11:55:03.678182 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:03.678145 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:55:03.678661 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:03.678197 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:55:03.690471 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:03.690445 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:55:04.127124 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:04.127096 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:55:18.396312 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.396275 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88"] Sep 15 11:55:18.396798 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.396533 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" podUID="ab6a02d4-4734-4f6b-a441-cec308808fce" containerName="payload-processing" containerID="cri-o://449353d83457ec4d513b1ac0e4d5baef276ebdf4f42198e81a38dc3af7769c74" gracePeriod=30 Sep 15 11:55:18.401936 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.401906 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl"] Sep 15 11:55:18.402175 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.402149 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" podUID="064fdeeb-c0ba-4c25-9246-052b10f9f3fc" containerName="payload-pre-processing" containerID="cri-o://e3b22f1b067519d86e8200bcf0a2b466798db287fbcfe6763755fabb02874c47" gracePeriod=30 Sep 15 11:55:18.646695 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.646632 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" Sep 15 11:55:18.673014 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.672986 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" Sep 15 11:55:18.720423 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.720394 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvczn\" (UniqueName: \"kubernetes.io/projected/064fdeeb-c0ba-4c25-9246-052b10f9f3fc-kube-api-access-lvczn\") pod \"064fdeeb-c0ba-4c25-9246-052b10f9f3fc\" (UID: \"064fdeeb-c0ba-4c25-9246-052b10f9f3fc\") " Sep 15 11:55:18.720599 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.720446 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6a02d4-4734-4f6b-a441-cec308808fce-plugins-config-volume\") pod \"ab6a02d4-4734-4f6b-a441-cec308808fce\" (UID: \"ab6a02d4-4734-4f6b-a441-cec308808fce\") " Sep 15 11:55:18.720599 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.720465 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064fdeeb-c0ba-4c25-9246-052b10f9f3fc-plugins-config-volume\") pod \"064fdeeb-c0ba-4c25-9246-052b10f9f3fc\" (UID: \"064fdeeb-c0ba-4c25-9246-052b10f9f3fc\") " Sep 15 11:55:18.720599 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.720534 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tgcp\" (UniqueName: \"kubernetes.io/projected/ab6a02d4-4734-4f6b-a441-cec308808fce-kube-api-access-5tgcp\") pod \"ab6a02d4-4734-4f6b-a441-cec308808fce\" (UID: \"ab6a02d4-4734-4f6b-a441-cec308808fce\") " Sep 15 11:55:18.720897 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.720870 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064fdeeb-c0ba-4c25-9246-052b10f9f3fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "064fdeeb-c0ba-4c25-9246-052b10f9f3fc" (UID: "064fdeeb-c0ba-4c25-9246-052b10f9f3fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:55:18.720991 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.720911 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab6a02d4-4734-4f6b-a441-cec308808fce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab6a02d4-4734-4f6b-a441-cec308808fce" (UID: "ab6a02d4-4734-4f6b-a441-cec308808fce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:55:18.722667 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.722644 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6a02d4-4734-4f6b-a441-cec308808fce-kube-api-access-5tgcp" (OuterVolumeSpecName: "kube-api-access-5tgcp") pod "ab6a02d4-4734-4f6b-a441-cec308808fce" (UID: "ab6a02d4-4734-4f6b-a441-cec308808fce"). InnerVolumeSpecName "kube-api-access-5tgcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:55:18.722729 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.722667 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064fdeeb-c0ba-4c25-9246-052b10f9f3fc-kube-api-access-lvczn" (OuterVolumeSpecName: "kube-api-access-lvczn") pod "064fdeeb-c0ba-4c25-9246-052b10f9f3fc" (UID: "064fdeeb-c0ba-4c25-9246-052b10f9f3fc"). InnerVolumeSpecName "kube-api-access-lvczn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:55:18.822067 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.822006 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tgcp\" (UniqueName: \"kubernetes.io/projected/ab6a02d4-4734-4f6b-a441-cec308808fce-kube-api-access-5tgcp\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:55:18.822067 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.822061 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvczn\" (UniqueName: \"kubernetes.io/projected/064fdeeb-c0ba-4c25-9246-052b10f9f3fc-kube-api-access-lvczn\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:55:18.822067 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.822074 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6a02d4-4734-4f6b-a441-cec308808fce-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:55:18.822302 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:18.822088 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064fdeeb-c0ba-4c25-9246-052b10f9f3fc-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:55:19.182886 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.182852 2573 generic.go:356] "Generic (PLEG): container finished" podID="ab6a02d4-4734-4f6b-a441-cec308808fce" containerID="449353d83457ec4d513b1ac0e4d5baef276ebdf4f42198e81a38dc3af7769c74" exitCode=0 Sep 15 11:55:19.183126 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.182898 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" Sep 15 11:55:19.183126 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.182947 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" event={"ID":"ab6a02d4-4734-4f6b-a441-cec308808fce","Type":"ContainerDied","Data":"449353d83457ec4d513b1ac0e4d5baef276ebdf4f42198e81a38dc3af7769c74"} Sep 15 11:55:19.183126 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.183012 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88" event={"ID":"ab6a02d4-4734-4f6b-a441-cec308808fce","Type":"ContainerDied","Data":"97fac44e659ffcee3e56876f628b275f6ca08474e822d04c3c7e12499fab78e0"} Sep 15 11:55:19.183126 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.183068 2573 scope.go:117] "RemoveContainer" containerID="449353d83457ec4d513b1ac0e4d5baef276ebdf4f42198e81a38dc3af7769c74" Sep 15 11:55:19.184366 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.184345 2573 generic.go:356] "Generic (PLEG): container finished" podID="064fdeeb-c0ba-4c25-9246-052b10f9f3fc" containerID="e3b22f1b067519d86e8200bcf0a2b466798db287fbcfe6763755fabb02874c47" exitCode=0 Sep 15 11:55:19.184460 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.184391 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" Sep 15 11:55:19.184460 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.184412 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" event={"ID":"064fdeeb-c0ba-4c25-9246-052b10f9f3fc","Type":"ContainerDied","Data":"e3b22f1b067519d86e8200bcf0a2b466798db287fbcfe6763755fabb02874c47"} Sep 15 11:55:19.184460 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.184450 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl" event={"ID":"064fdeeb-c0ba-4c25-9246-052b10f9f3fc","Type":"ContainerDied","Data":"f92302fb1e094015cca060c241d8d52820244883a2c81e3c53cb3afd8d8bf3d3"} Sep 15 11:55:19.191667 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.191648 2573 scope.go:117] "RemoveContainer" containerID="449353d83457ec4d513b1ac0e4d5baef276ebdf4f42198e81a38dc3af7769c74" Sep 15 11:55:19.191955 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:55:19.191937 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449353d83457ec4d513b1ac0e4d5baef276ebdf4f42198e81a38dc3af7769c74\": container with ID starting with 449353d83457ec4d513b1ac0e4d5baef276ebdf4f42198e81a38dc3af7769c74 not found: ID does not exist" containerID="449353d83457ec4d513b1ac0e4d5baef276ebdf4f42198e81a38dc3af7769c74" Sep 15 11:55:19.192013 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.191965 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449353d83457ec4d513b1ac0e4d5baef276ebdf4f42198e81a38dc3af7769c74"} err="failed to get container status \"449353d83457ec4d513b1ac0e4d5baef276ebdf4f42198e81a38dc3af7769c74\": rpc error: code = NotFound desc = could not find container \"449353d83457ec4d513b1ac0e4d5baef276ebdf4f42198e81a38dc3af7769c74\": container with ID starting with 449353d83457ec4d513b1ac0e4d5baef276ebdf4f42198e81a38dc3af7769c74 not found: ID does not exist" Sep 15 11:55:19.192013 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.191982 2573 scope.go:117] "RemoveContainer" containerID="e3b22f1b067519d86e8200bcf0a2b466798db287fbcfe6763755fabb02874c47" Sep 15 11:55:19.199955 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.199939 2573 scope.go:117] "RemoveContainer" containerID="e3b22f1b067519d86e8200bcf0a2b466798db287fbcfe6763755fabb02874c47" Sep 15 11:55:19.200227 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:55:19.200209 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b22f1b067519d86e8200bcf0a2b466798db287fbcfe6763755fabb02874c47\": container with ID starting with e3b22f1b067519d86e8200bcf0a2b466798db287fbcfe6763755fabb02874c47 not found: ID does not exist" containerID="e3b22f1b067519d86e8200bcf0a2b466798db287fbcfe6763755fabb02874c47" Sep 15 11:55:19.200286 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.200232 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b22f1b067519d86e8200bcf0a2b466798db287fbcfe6763755fabb02874c47"} err="failed to get container status \"e3b22f1b067519d86e8200bcf0a2b466798db287fbcfe6763755fabb02874c47\": rpc error: code = NotFound desc = could not find container \"e3b22f1b067519d86e8200bcf0a2b466798db287fbcfe6763755fabb02874c47\": container with ID starting with e3b22f1b067519d86e8200bcf0a2b466798db287fbcfe6763755fabb02874c47 not found: ID does not exist" Sep 15 11:55:19.208382 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.208359 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl"] Sep 15 11:55:19.212622 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.212602 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9064d2be-6f7485c98-l7lgl"] Sep 15 11:55:19.221924 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.221905 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88"] Sep 15 11:55:19.226484 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.226443 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9064d2be-cd7b4c7d9-v4s88"] Sep 15 11:55:19.237481 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.237454 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="064fdeeb-c0ba-4c25-9246-052b10f9f3fc" path="/var/lib/kubelet/pods/064fdeeb-c0ba-4c25-9246-052b10f9f3fc/volumes" Sep 15 11:55:19.237868 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:19.237849 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab6a02d4-4734-4f6b-a441-cec308808fce" path="/var/lib/kubelet/pods/ab6a02d4-4734-4f6b-a441-cec308808fce/volumes" Sep 15 11:55:44.739749 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.739713 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v"] Sep 15 11:55:44.740182 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.740089 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab6a02d4-4734-4f6b-a441-cec308808fce" containerName="payload-processing" Sep 15 11:55:44.740182 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.740102 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab6a02d4-4734-4f6b-a441-cec308808fce" containerName="payload-processing" Sep 15 11:55:44.740182 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.740111 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="064fdeeb-c0ba-4c25-9246-052b10f9f3fc" containerName="payload-pre-processing" Sep 15 11:55:44.740182 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.740116 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="064fdeeb-c0ba-4c25-9246-052b10f9f3fc" containerName="payload-pre-processing" Sep 15 11:55:44.740182 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.740178 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="064fdeeb-c0ba-4c25-9246-052b10f9f3fc" containerName="payload-pre-processing" Sep 15 11:55:44.740353 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.740187 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab6a02d4-4734-4f6b-a441-cec308808fce" containerName="payload-processing" Sep 15 11:55:44.743368 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.743351 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" Sep 15 11:55:44.745940 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.745911 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-7f9f55-dockercfg-2sl6g\"" Sep 15 11:55:44.746097 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.745945 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-7f9f55\"" Sep 15 11:55:44.752742 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.752721 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v"] Sep 15 11:55:44.799054 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.799013 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w"] Sep 15 11:55:44.802269 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.802254 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" Sep 15 11:55:44.810954 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.810930 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w"] Sep 15 11:55:44.850579 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.850548 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25341636-cbaf-4bfd-b040-c7e4bf118e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v\" (UID: \"25341636-cbaf-4bfd-b040-c7e4bf118e06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" Sep 15 11:55:44.850723 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.850611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rbxc\" (UniqueName: \"kubernetes.io/projected/25341636-cbaf-4bfd-b040-c7e4bf118e06-kube-api-access-6rbxc\") pod \"payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v\" (UID: \"25341636-cbaf-4bfd-b040-c7e4bf118e06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" Sep 15 11:55:44.952008 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.951976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz5vf\" (UniqueName: \"kubernetes.io/projected/0bc02592-5c04-4a91-8306-5fe1c540169a-kube-api-access-pz5vf\") pod \"payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w\" (UID: \"0bc02592-5c04-4a91-8306-5fe1c540169a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" Sep 15 11:55:44.952186 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.952042 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25341636-cbaf-4bfd-b040-c7e4bf118e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v\" (UID: \"25341636-cbaf-4bfd-b040-c7e4bf118e06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" Sep 15 11:55:44.952186 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.952143 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rbxc\" (UniqueName: \"kubernetes.io/projected/25341636-cbaf-4bfd-b040-c7e4bf118e06-kube-api-access-6rbxc\") pod \"payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v\" (UID: \"25341636-cbaf-4bfd-b040-c7e4bf118e06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" Sep 15 11:55:44.952266 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.952194 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc02592-5c04-4a91-8306-5fe1c540169a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w\" (UID: \"0bc02592-5c04-4a91-8306-5fe1c540169a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" Sep 15 11:55:44.952730 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.952711 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25341636-cbaf-4bfd-b040-c7e4bf118e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v\" (UID: \"25341636-cbaf-4bfd-b040-c7e4bf118e06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" Sep 15 11:55:44.960354 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:44.960330 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rbxc\" (UniqueName: \"kubernetes.io/projected/25341636-cbaf-4bfd-b040-c7e4bf118e06-kube-api-access-6rbxc\") pod \"payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v\" (UID: \"25341636-cbaf-4bfd-b040-c7e4bf118e06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" Sep 15 11:55:45.052619 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:45.052588 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc02592-5c04-4a91-8306-5fe1c540169a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w\" (UID: \"0bc02592-5c04-4a91-8306-5fe1c540169a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" Sep 15 11:55:45.052774 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:45.052623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz5vf\" (UniqueName: \"kubernetes.io/projected/0bc02592-5c04-4a91-8306-5fe1c540169a-kube-api-access-pz5vf\") pod \"payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w\" (UID: \"0bc02592-5c04-4a91-8306-5fe1c540169a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" Sep 15 11:55:45.053277 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:45.053257 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc02592-5c04-4a91-8306-5fe1c540169a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w\" (UID: \"0bc02592-5c04-4a91-8306-5fe1c540169a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" Sep 15 11:55:45.054469 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:45.054449 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" Sep 15 11:55:45.062739 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:45.062718 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz5vf\" (UniqueName: \"kubernetes.io/projected/0bc02592-5c04-4a91-8306-5fe1c540169a-kube-api-access-pz5vf\") pod \"payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w\" (UID: \"0bc02592-5c04-4a91-8306-5fe1c540169a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" Sep 15 11:55:45.113091 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:45.113059 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" Sep 15 11:55:45.326288 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:45.326260 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v"] Sep 15 11:55:45.328184 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:55:45.328157 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25341636_cbaf_4bfd_b040_c7e4bf118e06.slice/crio-b730ce09e2df242c33f7b0d6580efa4162d9ff9c1dbb5bbe77faf2ebde99563f WatchSource:0}: Error finding container b730ce09e2df242c33f7b0d6580efa4162d9ff9c1dbb5bbe77faf2ebde99563f: Status 404 returned error can't find the container with id b730ce09e2df242c33f7b0d6580efa4162d9ff9c1dbb5bbe77faf2ebde99563f Sep 15 11:55:45.389836 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:45.389810 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w"] Sep 15 11:55:45.390989 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:55:45.390965 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc02592_5c04_4a91_8306_5fe1c540169a.slice/crio-b3fb13fdbe7bffc0f8950b2285c97965c88f41192de4327a6630525819ed92b6 WatchSource:0}: Error finding container b3fb13fdbe7bffc0f8950b2285c97965c88f41192de4327a6630525819ed92b6: Status 404 returned error can't find the container with id b3fb13fdbe7bffc0f8950b2285c97965c88f41192de4327a6630525819ed92b6 Sep 15 11:55:46.310929 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:46.310892 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" event={"ID":"0bc02592-5c04-4a91-8306-5fe1c540169a","Type":"ContainerStarted","Data":"ca68b7da6e5a5bdd79566626e3b23d0fc4127349b8f3d1ab48f4f59fb7ace71f"} Sep 15 11:55:46.311407 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:46.310948 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" event={"ID":"0bc02592-5c04-4a91-8306-5fe1c540169a","Type":"ContainerStarted","Data":"b3fb13fdbe7bffc0f8950b2285c97965c88f41192de4327a6630525819ed92b6"} Sep 15 11:55:46.311407 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:46.310976 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" Sep 15 11:55:46.312505 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:46.312480 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" event={"ID":"25341636-cbaf-4bfd-b040-c7e4bf118e06","Type":"ContainerStarted","Data":"8511668b25b2b9e48a0acf66ea25d61c3144323ab75cb81f3a7213c06cb43dcd"} Sep 15 11:55:46.312634 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:46.312519 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" event={"ID":"25341636-cbaf-4bfd-b040-c7e4bf118e06","Type":"ContainerStarted","Data":"b730ce09e2df242c33f7b0d6580efa4162d9ff9c1dbb5bbe77faf2ebde99563f"} Sep 15 11:55:46.312634 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:46.312550 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" Sep 15 11:55:46.329336 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:46.329282 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" podStartSLOduration=2.329267133 podStartE2EDuration="2.329267133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:55:46.32714006 +0000 UTC m=+1133.557066608" watchObservedRunningTime="2026-09-15 11:55:46.329267133 +0000 UTC m=+1133.559193669" Sep 15 11:55:46.344487 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:46.344436 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" podStartSLOduration=2.344418048 podStartE2EDuration="2.344418048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:55:46.343549907 +0000 UTC m=+1133.573476445" watchObservedRunningTime="2026-09-15 11:55:46.344418048 +0000 UTC m=+1133.574344595" Sep 15 11:55:57.320000 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:57.319967 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" Sep 15 11:55:57.320398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:55:57.320059 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" Sep 15 11:56:14.919838 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:14.919802 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn"] Sep 15 11:56:14.924643 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:14.924626 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" Sep 15 11:56:14.927064 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:14.927042 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-d2de76-dockercfg-zhjtp\"" Sep 15 11:56:14.927179 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:14.927047 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-d2de76\"" Sep 15 11:56:14.931813 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:14.931783 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn"] Sep 15 11:56:14.954313 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:14.954280 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5"] Sep 15 11:56:14.957566 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:14.957548 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" Sep 15 11:56:14.966108 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:14.966081 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5"] Sep 15 11:56:15.014220 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.014179 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q47fx\" (UniqueName: \"kubernetes.io/projected/8d61f56c-a69c-439a-950c-d40dbf997396-kube-api-access-q47fx\") pod \"payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5\" (UID: \"8d61f56c-a69c-439a-950c-d40dbf997396\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" Sep 15 11:56:15.014405 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.014230 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a680930-2cc1-41d4-833d-4eca26a6f52a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn\" (UID: \"2a680930-2cc1-41d4-833d-4eca26a6f52a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" Sep 15 11:56:15.014405 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.014342 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6w7q\" (UniqueName: \"kubernetes.io/projected/2a680930-2cc1-41d4-833d-4eca26a6f52a-kube-api-access-j6w7q\") pod \"payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn\" (UID: \"2a680930-2cc1-41d4-833d-4eca26a6f52a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" Sep 15 11:56:15.014488 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.014417 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d61f56c-a69c-439a-950c-d40dbf997396-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5\" (UID: \"8d61f56c-a69c-439a-950c-d40dbf997396\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" Sep 15 11:56:15.115379 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.115332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q47fx\" (UniqueName: \"kubernetes.io/projected/8d61f56c-a69c-439a-950c-d40dbf997396-kube-api-access-q47fx\") pod \"payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5\" (UID: \"8d61f56c-a69c-439a-950c-d40dbf997396\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" Sep 15 11:56:15.115582 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.115383 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a680930-2cc1-41d4-833d-4eca26a6f52a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn\" (UID: \"2a680930-2cc1-41d4-833d-4eca26a6f52a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" Sep 15 11:56:15.115582 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.115429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6w7q\" (UniqueName: \"kubernetes.io/projected/2a680930-2cc1-41d4-833d-4eca26a6f52a-kube-api-access-j6w7q\") pod \"payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn\" (UID: \"2a680930-2cc1-41d4-833d-4eca26a6f52a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" Sep 15 11:56:15.115582 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.115497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d61f56c-a69c-439a-950c-d40dbf997396-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5\" (UID: \"8d61f56c-a69c-439a-950c-d40dbf997396\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" Sep 15 11:56:15.116297 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.116275 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a680930-2cc1-41d4-833d-4eca26a6f52a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn\" (UID: \"2a680930-2cc1-41d4-833d-4eca26a6f52a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" Sep 15 11:56:15.116297 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.116291 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d61f56c-a69c-439a-950c-d40dbf997396-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5\" (UID: \"8d61f56c-a69c-439a-950c-d40dbf997396\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" Sep 15 11:56:15.124836 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.124812 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6w7q\" (UniqueName: \"kubernetes.io/projected/2a680930-2cc1-41d4-833d-4eca26a6f52a-kube-api-access-j6w7q\") pod \"payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn\" (UID: \"2a680930-2cc1-41d4-833d-4eca26a6f52a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" Sep 15 11:56:15.124949 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.124845 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q47fx\" (UniqueName: \"kubernetes.io/projected/8d61f56c-a69c-439a-950c-d40dbf997396-kube-api-access-q47fx\") pod \"payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5\" (UID: \"8d61f56c-a69c-439a-950c-d40dbf997396\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" Sep 15 11:56:15.236050 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.235958 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" Sep 15 11:56:15.268278 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.268245 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" Sep 15 11:56:15.507092 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.507067 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn"] Sep 15 11:56:15.508977 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:56:15.508951 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a680930_2cc1_41d4_833d_4eca26a6f52a.slice/crio-a420aeec3a7d0698d37128fdc6f9560aafb6e3b596a85e0e1e5c36040915b2de WatchSource:0}: Error finding container a420aeec3a7d0698d37128fdc6f9560aafb6e3b596a85e0e1e5c36040915b2de: Status 404 returned error can't find the container with id a420aeec3a7d0698d37128fdc6f9560aafb6e3b596a85e0e1e5c36040915b2de Sep 15 11:56:15.562363 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:15.562335 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5"] Sep 15 11:56:15.572164 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:56:15.572141 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d61f56c_a69c_439a_950c_d40dbf997396.slice/crio-74a3b7bc49fef80193a9aa89d5a0c4656f0f3bbd134bbaecebe881ef8fa34b04 WatchSource:0}: Error finding container 74a3b7bc49fef80193a9aa89d5a0c4656f0f3bbd134bbaecebe881ef8fa34b04: Status 404 returned error can't find the container with id 74a3b7bc49fef80193a9aa89d5a0c4656f0f3bbd134bbaecebe881ef8fa34b04 Sep 15 11:56:16.452980 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:16.452941 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" event={"ID":"2a680930-2cc1-41d4-833d-4eca26a6f52a","Type":"ContainerStarted","Data":"3b1868b2d0a1bc76be2c8f9ce7ed5612e19e5444a5ed09db1b7ecde32edcbbcf"} Sep 15 11:56:16.453505 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:16.453017 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" event={"ID":"2a680930-2cc1-41d4-833d-4eca26a6f52a","Type":"ContainerStarted","Data":"a420aeec3a7d0698d37128fdc6f9560aafb6e3b596a85e0e1e5c36040915b2de"} Sep 15 11:56:16.453505 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:16.453154 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" Sep 15 11:56:16.454653 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:16.454629 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" event={"ID":"8d61f56c-a69c-439a-950c-d40dbf997396","Type":"ContainerStarted","Data":"14543f644b23a4dcbd92ee0d266124ff30135472c26a74886babc5fde2ec7190"} Sep 15 11:56:16.454754 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:16.454674 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" event={"ID":"8d61f56c-a69c-439a-950c-d40dbf997396","Type":"ContainerStarted","Data":"74a3b7bc49fef80193a9aa89d5a0c4656f0f3bbd134bbaecebe881ef8fa34b04"} Sep 15 11:56:16.454754 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:16.454712 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" Sep 15 11:56:16.468645 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:16.468583 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" podStartSLOduration=2.468569997 podStartE2EDuration="2.468569997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:16.468270426 +0000 UTC m=+1163.698196965" watchObservedRunningTime="2026-09-15 11:56:16.468569997 +0000 UTC m=+1163.698496533" Sep 15 11:56:16.486284 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:16.486232 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" podStartSLOduration=2.4862132040000002 podStartE2EDuration="2.486213204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:16.484221337 +0000 UTC m=+1163.714147872" watchObservedRunningTime="2026-09-15 11:56:16.486213204 +0000 UTC m=+1163.716139741" Sep 15 11:56:27.461254 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:27.461223 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" Sep 15 11:56:27.461694 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:27.461448 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" Sep 15 11:56:37.755869 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:37.755830 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7092-933059"] Sep 15 11:56:37.759301 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:37.759284 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-933059" Sep 15 11:56:37.762498 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:37.762475 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dg4tz\"" Sep 15 11:56:37.764628 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:37.764602 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-933059"] Sep 15 11:56:37.921395 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:37.921362 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5h2n\" (UniqueName: \"kubernetes.io/projected/a7da02e0-862b-416f-9fe3-700d381fcb81-kube-api-access-m5h2n\") pod \"test-curl-7092-933059\" (UID: \"a7da02e0-862b-416f-9fe3-700d381fcb81\") " pod="opendatahub/test-curl-7092-933059" Sep 15 11:56:38.022757 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:38.022669 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5h2n\" (UniqueName: \"kubernetes.io/projected/a7da02e0-862b-416f-9fe3-700d381fcb81-kube-api-access-m5h2n\") pod \"test-curl-7092-933059\" (UID: \"a7da02e0-862b-416f-9fe3-700d381fcb81\") " pod="opendatahub/test-curl-7092-933059" Sep 15 11:56:38.034339 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:38.034315 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5h2n\" (UniqueName: \"kubernetes.io/projected/a7da02e0-862b-416f-9fe3-700d381fcb81-kube-api-access-m5h2n\") pod \"test-curl-7092-933059\" (UID: \"a7da02e0-862b-416f-9fe3-700d381fcb81\") " pod="opendatahub/test-curl-7092-933059" Sep 15 11:56:38.071307 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:38.071281 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-933059" Sep 15 11:56:38.344767 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:38.344641 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-933059"] Sep 15 11:56:38.346924 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:56:38.346899 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7da02e0_862b_416f_9fe3_700d381fcb81.slice/crio-78471b4461560860369eeefc6df9b9eba3365894e27a86af71a43100427cb9fa WatchSource:0}: Error finding container 78471b4461560860369eeefc6df9b9eba3365894e27a86af71a43100427cb9fa: Status 404 returned error can't find the container with id 78471b4461560860369eeefc6df9b9eba3365894e27a86af71a43100427cb9fa Sep 15 11:56:38.555075 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:38.555016 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-933059" event={"ID":"a7da02e0-862b-416f-9fe3-700d381fcb81","Type":"ContainerStarted","Data":"591916d4d5a1ed32f44f2a90ac6ed513ec402a53cd0b095f56cd9818a77bc045"} Sep 15 11:56:38.555247 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:38.555083 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-933059" event={"ID":"a7da02e0-862b-416f-9fe3-700d381fcb81","Type":"ContainerStarted","Data":"78471b4461560860369eeefc6df9b9eba3365894e27a86af71a43100427cb9fa"} Sep 15 11:56:38.571975 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:38.571929 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7092-933059" podStartSLOduration=1.571910642 podStartE2EDuration="1.571910642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:38.569626989 +0000 UTC m=+1185.799553526" watchObservedRunningTime="2026-09-15 11:56:38.571910642 +0000 UTC m=+1185.801837178" Sep 15 11:56:39.213393 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:39.213355 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7092-933059"] Sep 15 11:56:39.219343 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:39.219318 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7092-933059"] Sep 15 11:56:39.346593 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:39.346553 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7092-0b5a65"] Sep 15 11:56:39.346978 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:39.346963 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7da02e0-862b-416f-9fe3-700d381fcb81" containerName="test-curl-7092-933059" Sep 15 11:56:39.347044 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:39.346980 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7da02e0-862b-416f-9fe3-700d381fcb81" containerName="test-curl-7092-933059" Sep 15 11:56:39.347085 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:39.347074 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7da02e0-862b-416f-9fe3-700d381fcb81" containerName="test-curl-7092-933059" Sep 15 11:56:39.350096 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:39.350080 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-0b5a65" Sep 15 11:56:39.359207 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:39.359178 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-0b5a65"] Sep 15 11:56:39.536231 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:39.536198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ggs6\" (UniqueName: \"kubernetes.io/projected/59eb1018-efcc-4a30-a623-e34ff053fb35-kube-api-access-4ggs6\") pod \"test-curl-7092-0b5a65\" (UID: \"59eb1018-efcc-4a30-a623-e34ff053fb35\") " pod="opendatahub/test-curl-7092-0b5a65" Sep 15 11:56:39.559681 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:56:39.559655 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7092-933059" container="test-curl-7092-933059" Sep 15 11:56:39.637282 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:39.637200 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ggs6\" (UniqueName: \"kubernetes.io/projected/59eb1018-efcc-4a30-a623-e34ff053fb35-kube-api-access-4ggs6\") pod \"test-curl-7092-0b5a65\" (UID: \"59eb1018-efcc-4a30-a623-e34ff053fb35\") " pod="opendatahub/test-curl-7092-0b5a65" Sep 15 11:56:39.645567 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:39.645548 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ggs6\" (UniqueName: \"kubernetes.io/projected/59eb1018-efcc-4a30-a623-e34ff053fb35-kube-api-access-4ggs6\") pod \"test-curl-7092-0b5a65\" (UID: \"59eb1018-efcc-4a30-a623-e34ff053fb35\") " pod="opendatahub/test-curl-7092-0b5a65" Sep 15 11:56:39.660515 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:39.660495 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-0b5a65" Sep 15 11:56:39.941462 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:39.941433 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-0b5a65"] Sep 15 11:56:39.942596 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:56:39.942567 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59eb1018_efcc_4a30_a623_e34ff053fb35.slice/crio-22d3456c7b90fc946a5e6d2f558d69c1f55df2a66dba976662661ffecaeab12b WatchSource:0}: Error finding container 22d3456c7b90fc946a5e6d2f558d69c1f55df2a66dba976662661ffecaeab12b: Status 404 returned error can't find the container with id 22d3456c7b90fc946a5e6d2f558d69c1f55df2a66dba976662661ffecaeab12b Sep 15 11:56:40.571982 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:40.571937 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-0b5a65" event={"ID":"59eb1018-efcc-4a30-a623-e34ff053fb35","Type":"ContainerStarted","Data":"3e6fe73b0e881a437bb242be7d45f0e451c38cb305e1091cca9074d21e1c26ed"} Sep 15 11:56:40.571982 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:40.571986 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-0b5a65" event={"ID":"59eb1018-efcc-4a30-a623-e34ff053fb35","Type":"ContainerStarted","Data":"22d3456c7b90fc946a5e6d2f558d69c1f55df2a66dba976662661ffecaeab12b"} Sep 15 11:56:40.572499 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:40.572092 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7092-933059" podUID="a7da02e0-862b-416f-9fe3-700d381fcb81" containerName="test-curl-7092-933059" containerID="cri-o://591916d4d5a1ed32f44f2a90ac6ed513ec402a53cd0b095f56cd9818a77bc045" gracePeriod=2 Sep 15 11:56:40.588983 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:40.588933 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7092-0b5a65" podStartSLOduration=1.588916473 podStartE2EDuration="1.588916473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:40.588712824 +0000 UTC m=+1187.818639360" watchObservedRunningTime="2026-09-15 11:56:40.588916473 +0000 UTC m=+1187.818843012" Sep 15 11:56:41.217012 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:41.216977 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7092-0b5a65"] Sep 15 11:56:41.221122 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:41.221095 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7092-0b5a65"] Sep 15 11:56:41.349875 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:41.349831 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7092-9b6664"] Sep 15 11:56:41.350317 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:41.350296 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59eb1018-efcc-4a30-a623-e34ff053fb35" containerName="test-curl-7092-0b5a65" Sep 15 11:56:41.350418 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:41.350320 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="59eb1018-efcc-4a30-a623-e34ff053fb35" containerName="test-curl-7092-0b5a65" Sep 15 11:56:41.350418 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:41.350406 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="59eb1018-efcc-4a30-a623-e34ff053fb35" containerName="test-curl-7092-0b5a65" Sep 15 11:56:41.353636 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:41.353613 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-9b6664" Sep 15 11:56:41.361610 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:41.361048 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-9b6664"] Sep 15 11:56:41.454665 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:41.454634 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwbmj\" (UniqueName: \"kubernetes.io/projected/b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5-kube-api-access-wwbmj\") pod \"test-curl-7092-9b6664\" (UID: \"b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5\") " pod="opendatahub/test-curl-7092-9b6664" Sep 15 11:56:41.555155 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:41.555118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwbmj\" (UniqueName: \"kubernetes.io/projected/b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5-kube-api-access-wwbmj\") pod \"test-curl-7092-9b6664\" (UID: \"b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5\") " pod="opendatahub/test-curl-7092-9b6664" Sep 15 11:56:41.563097 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:41.563068 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwbmj\" (UniqueName: \"kubernetes.io/projected/b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5-kube-api-access-wwbmj\") pod \"test-curl-7092-9b6664\" (UID: \"b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5\") " pod="opendatahub/test-curl-7092-9b6664" Sep 15 11:56:41.576551 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:56:41.576527 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7092-0b5a65" container="test-curl-7092-0b5a65" Sep 15 11:56:41.664213 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:41.664136 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-9b6664" Sep 15 11:56:41.932563 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:41.932491 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-9b6664"] Sep 15 11:56:41.935469 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:56:41.935442 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb68e2eca_b5ef_4bd6_9fa6_da1218ede3b5.slice/crio-e94ca807a1b31c478858e0fb9556c6677f906fd13d6bc5afd3d2687ec490415b WatchSource:0}: Error finding container e94ca807a1b31c478858e0fb9556c6677f906fd13d6bc5afd3d2687ec490415b: Status 404 returned error can't find the container with id e94ca807a1b31c478858e0fb9556c6677f906fd13d6bc5afd3d2687ec490415b Sep 15 11:56:42.583396 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:42.583356 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-9b6664" event={"ID":"b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5","Type":"ContainerStarted","Data":"8b27bd70cf7d648bd23552f4ec36ecbbd851487b19f3bb5867ec298972fc04f2"} Sep 15 11:56:42.583766 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:42.583410 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-9b6664" event={"ID":"b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5","Type":"ContainerStarted","Data":"e94ca807a1b31c478858e0fb9556c6677f906fd13d6bc5afd3d2687ec490415b"} Sep 15 11:56:42.583766 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:42.583472 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7092-0b5a65" podUID="59eb1018-efcc-4a30-a623-e34ff053fb35" containerName="test-curl-7092-0b5a65" containerID="cri-o://3e6fe73b0e881a437bb242be7d45f0e451c38cb305e1091cca9074d21e1c26ed" gracePeriod=2 Sep 15 11:56:42.596944 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:42.596902 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7092-9b6664" podStartSLOduration=1.59688483 podStartE2EDuration="1.59688483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:42.596530298 +0000 UTC m=+1189.826456835" watchObservedRunningTime="2026-09-15 11:56:42.59688483 +0000 UTC m=+1189.826811367" Sep 15 11:56:42.767909 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:42.767885 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-933059" Sep 15 11:56:42.866844 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:42.866759 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5h2n\" (UniqueName: \"kubernetes.io/projected/a7da02e0-862b-416f-9fe3-700d381fcb81-kube-api-access-m5h2n\") pod \"a7da02e0-862b-416f-9fe3-700d381fcb81\" (UID: \"a7da02e0-862b-416f-9fe3-700d381fcb81\") " Sep 15 11:56:42.868725 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:42.868700 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7da02e0-862b-416f-9fe3-700d381fcb81-kube-api-access-m5h2n" (OuterVolumeSpecName: "kube-api-access-m5h2n") pod "a7da02e0-862b-416f-9fe3-700d381fcb81" (UID: "a7da02e0-862b-416f-9fe3-700d381fcb81"). InnerVolumeSpecName "kube-api-access-m5h2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:56:42.967678 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:42.967645 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5h2n\" (UniqueName: \"kubernetes.io/projected/a7da02e0-862b-416f-9fe3-700d381fcb81-kube-api-access-m5h2n\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:56:43.177230 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.177137 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7092-9b6664"] Sep 15 11:56:43.180573 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.180540 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7092-9b6664"] Sep 15 11:56:43.234925 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.234893 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7da02e0-862b-416f-9fe3-700d381fcb81" path="/var/lib/kubelet/pods/a7da02e0-862b-416f-9fe3-700d381fcb81/volumes" Sep 15 11:56:43.311079 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.311014 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7092-9051d0"] Sep 15 11:56:43.311436 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.311420 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5" containerName="test-curl-7092-9b6664" Sep 15 11:56:43.311436 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.311436 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5" containerName="test-curl-7092-9b6664" Sep 15 11:56:43.311543 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.311498 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5" containerName="test-curl-7092-9b6664" Sep 15 11:56:43.314660 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.314637 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-9051d0"] Sep 15 11:56:43.314796 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.314722 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-9051d0" Sep 15 11:56:43.472697 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.472589 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sk48\" (UniqueName: \"kubernetes.io/projected/236a4fb3-568e-418f-9dc4-75241f439daf-kube-api-access-5sk48\") pod \"test-curl-7092-9051d0\" (UID: \"236a4fb3-568e-418f-9dc4-75241f439daf\") " pod="opendatahub/test-curl-7092-9051d0" Sep 15 11:56:43.573930 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.573881 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sk48\" (UniqueName: \"kubernetes.io/projected/236a4fb3-568e-418f-9dc4-75241f439daf-kube-api-access-5sk48\") pod \"test-curl-7092-9051d0\" (UID: \"236a4fb3-568e-418f-9dc4-75241f439daf\") " pod="opendatahub/test-curl-7092-9051d0" Sep 15 11:56:43.582124 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.582098 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sk48\" (UniqueName: \"kubernetes.io/projected/236a4fb3-568e-418f-9dc4-75241f439daf-kube-api-access-5sk48\") pod \"test-curl-7092-9051d0\" (UID: \"236a4fb3-568e-418f-9dc4-75241f439daf\") " pod="opendatahub/test-curl-7092-9051d0" Sep 15 11:56:43.590733 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.590705 2573 generic.go:356] "Generic (PLEG): container finished" podID="a7da02e0-862b-416f-9fe3-700d381fcb81" containerID="591916d4d5a1ed32f44f2a90ac6ed513ec402a53cd0b095f56cd9818a77bc045" exitCode=137 Sep 15 11:56:43.591146 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.590770 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-933059" Sep 15 11:56:43.591146 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.590783 2573 scope.go:117] "RemoveContainer" containerID="591916d4d5a1ed32f44f2a90ac6ed513ec402a53cd0b095f56cd9818a77bc045" Sep 15 11:56:43.591146 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:56:43.590980 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7092-9b6664" container="test-curl-7092-9b6664" Sep 15 11:56:43.598907 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.598886 2573 scope.go:117] "RemoveContainer" containerID="591916d4d5a1ed32f44f2a90ac6ed513ec402a53cd0b095f56cd9818a77bc045" Sep 15 11:56:43.599279 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:56:43.599256 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591916d4d5a1ed32f44f2a90ac6ed513ec402a53cd0b095f56cd9818a77bc045\": container with ID starting with 591916d4d5a1ed32f44f2a90ac6ed513ec402a53cd0b095f56cd9818a77bc045 not found: ID does not exist" containerID="591916d4d5a1ed32f44f2a90ac6ed513ec402a53cd0b095f56cd9818a77bc045" Sep 15 11:56:43.599356 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.599285 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591916d4d5a1ed32f44f2a90ac6ed513ec402a53cd0b095f56cd9818a77bc045"} err="failed to get container status \"591916d4d5a1ed32f44f2a90ac6ed513ec402a53cd0b095f56cd9818a77bc045\": rpc error: code = NotFound desc = could not find container \"591916d4d5a1ed32f44f2a90ac6ed513ec402a53cd0b095f56cd9818a77bc045\": container with ID starting with 591916d4d5a1ed32f44f2a90ac6ed513ec402a53cd0b095f56cd9818a77bc045 not found: ID does not exist" Sep 15 11:56:43.624560 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.624538 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-9051d0" Sep 15 11:56:43.893303 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:43.893281 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-9051d0"] Sep 15 11:56:43.895283 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:56:43.895250 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod236a4fb3_568e_418f_9dc4_75241f439daf.slice/crio-ba98607b7f8e32d31197810f8fe55607fbb029e6731f5bbc04d1690b6bc4ed32 WatchSource:0}: Error finding container ba98607b7f8e32d31197810f8fe55607fbb029e6731f5bbc04d1690b6bc4ed32: Status 404 returned error can't find the container with id ba98607b7f8e32d31197810f8fe55607fbb029e6731f5bbc04d1690b6bc4ed32 Sep 15 11:56:44.597661 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:44.597631 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-9051d0" event={"ID":"236a4fb3-568e-418f-9dc4-75241f439daf","Type":"ContainerStarted","Data":"dbfc457301621928a61dddbb5b830480bcecb8540c0a300d948801b5430def50"} Sep 15 11:56:44.598004 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:44.597672 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-9051d0" event={"ID":"236a4fb3-568e-418f-9dc4-75241f439daf","Type":"ContainerStarted","Data":"ba98607b7f8e32d31197810f8fe55607fbb029e6731f5bbc04d1690b6bc4ed32"} Sep 15 11:56:44.598767 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:44.598725 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7092-9b6664" podUID="b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5" containerName="test-curl-7092-9b6664" containerID="cri-o://8b27bd70cf7d648bd23552f4ec36ecbbd851487b19f3bb5867ec298972fc04f2" gracePeriod=2 Sep 15 11:56:44.612137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:44.612095 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7092-9051d0" podStartSLOduration=1.612079421 podStartE2EDuration="1.612079421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:44.611646909 +0000 UTC m=+1191.841573446" watchObservedRunningTime="2026-09-15 11:56:44.612079421 +0000 UTC m=+1191.842005956" Sep 15 11:56:44.757324 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:44.757293 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-0b5a65" Sep 15 11:56:44.783161 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:44.783135 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ggs6\" (UniqueName: \"kubernetes.io/projected/59eb1018-efcc-4a30-a623-e34ff053fb35-kube-api-access-4ggs6\") pod \"59eb1018-efcc-4a30-a623-e34ff053fb35\" (UID: \"59eb1018-efcc-4a30-a623-e34ff053fb35\") " Sep 15 11:56:44.785046 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:44.785011 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59eb1018-efcc-4a30-a623-e34ff053fb35-kube-api-access-4ggs6" (OuterVolumeSpecName: "kube-api-access-4ggs6") pod "59eb1018-efcc-4a30-a623-e34ff053fb35" (UID: "59eb1018-efcc-4a30-a623-e34ff053fb35"). InnerVolumeSpecName "kube-api-access-4ggs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:56:44.884300 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:44.884231 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ggs6\" (UniqueName: \"kubernetes.io/projected/59eb1018-efcc-4a30-a623-e34ff053fb35-kube-api-access-4ggs6\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:56:45.235196 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:45.235118 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59eb1018-efcc-4a30-a623-e34ff053fb35" path="/var/lib/kubelet/pods/59eb1018-efcc-4a30-a623-e34ff053fb35/volumes" Sep 15 11:56:45.273932 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:45.273906 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7092-9051d0"] Sep 15 11:56:45.279518 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:45.279496 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7092-9051d0"] Sep 15 11:56:45.604758 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:45.604723 2573 generic.go:356] "Generic (PLEG): container finished" podID="59eb1018-efcc-4a30-a623-e34ff053fb35" containerID="3e6fe73b0e881a437bb242be7d45f0e451c38cb305e1091cca9074d21e1c26ed" exitCode=137 Sep 15 11:56:45.604758 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:45.604760 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-0b5a65" Sep 15 11:56:45.605246 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:45.604820 2573 scope.go:117] "RemoveContainer" containerID="3e6fe73b0e881a437bb242be7d45f0e451c38cb305e1091cca9074d21e1c26ed" Sep 15 11:56:45.605246 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:56:45.605018 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7092-9051d0" container="test-curl-7092-9051d0" Sep 15 11:56:45.613350 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:45.613334 2573 scope.go:117] "RemoveContainer" containerID="3e6fe73b0e881a437bb242be7d45f0e451c38cb305e1091cca9074d21e1c26ed" Sep 15 11:56:45.613626 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:56:45.613607 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6fe73b0e881a437bb242be7d45f0e451c38cb305e1091cca9074d21e1c26ed\": container with ID starting with 3e6fe73b0e881a437bb242be7d45f0e451c38cb305e1091cca9074d21e1c26ed not found: ID does not exist" containerID="3e6fe73b0e881a437bb242be7d45f0e451c38cb305e1091cca9074d21e1c26ed" Sep 15 11:56:45.613676 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:45.613635 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6fe73b0e881a437bb242be7d45f0e451c38cb305e1091cca9074d21e1c26ed"} err="failed to get container status \"3e6fe73b0e881a437bb242be7d45f0e451c38cb305e1091cca9074d21e1c26ed\": rpc error: code = NotFound desc = could not find container \"3e6fe73b0e881a437bb242be7d45f0e451c38cb305e1091cca9074d21e1c26ed\": container with ID starting with 3e6fe73b0e881a437bb242be7d45f0e451c38cb305e1091cca9074d21e1c26ed not found: ID does not exist" Sep 15 11:56:46.610839 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:46.610804 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7092-9051d0" podUID="236a4fb3-568e-418f-9dc4-75241f439daf" containerName="test-curl-7092-9051d0" containerID="cri-o://dbfc457301621928a61dddbb5b830480bcecb8540c0a300d948801b5430def50" gracePeriod=2 Sep 15 11:56:46.613280 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:56:46.613234 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7092-9051d0\" is forbidden: User \"system:node:ip-10-0-128-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-189.ec2.internal' and this object" podUID="236a4fb3-568e-418f-9dc4-75241f439daf" pod="opendatahub/test-curl-7092-9051d0" Sep 15 11:56:46.772222 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:46.772199 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-9b6664" Sep 15 11:56:46.774505 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:56:46.774480 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7092-9051d0\" is forbidden: User \"system:node:ip-10-0-128-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-189.ec2.internal' and this object" podUID="236a4fb3-568e-418f-9dc4-75241f439daf" pod="opendatahub/test-curl-7092-9051d0" Sep 15 11:56:46.798298 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:46.798279 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwbmj\" (UniqueName: \"kubernetes.io/projected/b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5-kube-api-access-wwbmj\") pod \"b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5\" (UID: \"b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5\") " Sep 15 11:56:46.800254 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:46.800233 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5-kube-api-access-wwbmj" (OuterVolumeSpecName: "kube-api-access-wwbmj") pod "b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5" (UID: "b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5"). InnerVolumeSpecName "kube-api-access-wwbmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:56:46.899577 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:46.899501 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwbmj\" (UniqueName: \"kubernetes.io/projected/b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5-kube-api-access-wwbmj\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:56:47.235225 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:47.235148 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5" path="/var/lib/kubelet/pods/b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5/volumes" Sep 15 11:56:47.617016 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:47.616983 2573 generic.go:356] "Generic (PLEG): container finished" podID="b68e2eca-b5ef-4bd6-9fa6-da1218ede3b5" containerID="8b27bd70cf7d648bd23552f4ec36ecbbd851487b19f3bb5867ec298972fc04f2" exitCode=137 Sep 15 11:56:47.617482 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:47.617050 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-9b6664" Sep 15 11:56:47.617482 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:47.617075 2573 scope.go:117] "RemoveContainer" containerID="8b27bd70cf7d648bd23552f4ec36ecbbd851487b19f3bb5867ec298972fc04f2" Sep 15 11:56:47.625634 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:47.625624 2573 scope.go:117] "RemoveContainer" containerID="8b27bd70cf7d648bd23552f4ec36ecbbd851487b19f3bb5867ec298972fc04f2" Sep 15 11:56:47.625949 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:56:47.625929 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b27bd70cf7d648bd23552f4ec36ecbbd851487b19f3bb5867ec298972fc04f2\": container with ID starting with 8b27bd70cf7d648bd23552f4ec36ecbbd851487b19f3bb5867ec298972fc04f2 not found: ID does not exist" containerID="8b27bd70cf7d648bd23552f4ec36ecbbd851487b19f3bb5867ec298972fc04f2" Sep 15 11:56:47.626046 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:47.625957 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b27bd70cf7d648bd23552f4ec36ecbbd851487b19f3bb5867ec298972fc04f2"} err="failed to get container status \"8b27bd70cf7d648bd23552f4ec36ecbbd851487b19f3bb5867ec298972fc04f2\": rpc error: code = NotFound desc = could not find container \"8b27bd70cf7d648bd23552f4ec36ecbbd851487b19f3bb5867ec298972fc04f2\": container with ID starting with 8b27bd70cf7d648bd23552f4ec36ecbbd851487b19f3bb5867ec298972fc04f2 not found: ID does not exist" Sep 15 11:56:48.797072 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:48.797044 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-9051d0" Sep 15 11:56:48.816295 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:48.816269 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sk48\" (UniqueName: \"kubernetes.io/projected/236a4fb3-568e-418f-9dc4-75241f439daf-kube-api-access-5sk48\") pod \"236a4fb3-568e-418f-9dc4-75241f439daf\" (UID: \"236a4fb3-568e-418f-9dc4-75241f439daf\") " Sep 15 11:56:48.818137 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:48.818112 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236a4fb3-568e-418f-9dc4-75241f439daf-kube-api-access-5sk48" (OuterVolumeSpecName: "kube-api-access-5sk48") pod "236a4fb3-568e-418f-9dc4-75241f439daf" (UID: "236a4fb3-568e-418f-9dc4-75241f439daf"). InnerVolumeSpecName "kube-api-access-5sk48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:56:48.917104 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:48.917010 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sk48\" (UniqueName: \"kubernetes.io/projected/236a4fb3-568e-418f-9dc4-75241f439daf-kube-api-access-5sk48\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:56:49.235554 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:49.235479 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="236a4fb3-568e-418f-9dc4-75241f439daf" path="/var/lib/kubelet/pods/236a4fb3-568e-418f-9dc4-75241f439daf/volumes" Sep 15 11:56:49.630319 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:49.630282 2573 generic.go:356] "Generic (PLEG): container finished" podID="236a4fb3-568e-418f-9dc4-75241f439daf" containerID="dbfc457301621928a61dddbb5b830480bcecb8540c0a300d948801b5430def50" exitCode=137 Sep 15 11:56:49.630495 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:49.630327 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-9051d0" Sep 15 11:56:49.630495 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:49.630380 2573 scope.go:117] "RemoveContainer" containerID="dbfc457301621928a61dddbb5b830480bcecb8540c0a300d948801b5430def50" Sep 15 11:56:49.638694 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:49.638651 2573 scope.go:117] "RemoveContainer" containerID="dbfc457301621928a61dddbb5b830480bcecb8540c0a300d948801b5430def50" Sep 15 11:56:49.638987 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:56:49.638967 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbfc457301621928a61dddbb5b830480bcecb8540c0a300d948801b5430def50\": container with ID starting with dbfc457301621928a61dddbb5b830480bcecb8540c0a300d948801b5430def50 not found: ID does not exist" containerID="dbfc457301621928a61dddbb5b830480bcecb8540c0a300d948801b5430def50" Sep 15 11:56:49.639058 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:49.638996 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbfc457301621928a61dddbb5b830480bcecb8540c0a300d948801b5430def50"} err="failed to get container status \"dbfc457301621928a61dddbb5b830480bcecb8540c0a300d948801b5430def50\": rpc error: code = NotFound desc = could not find container \"dbfc457301621928a61dddbb5b830480bcecb8540c0a300d948801b5430def50\": container with ID starting with dbfc457301621928a61dddbb5b830480bcecb8540c0a300d948801b5430def50 not found: ID does not exist" Sep 15 11:56:53.268295 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:53.268264 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 11:56:53.270762 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:53.270738 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 11:56:53.275993 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:53.275972 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 11:56:53.281574 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:56:53.281556 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 11:57:14.830887 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:14.830854 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-d1e2fb"] Sep 15 11:57:14.833344 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:14.831238 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="236a4fb3-568e-418f-9dc4-75241f439daf" containerName="test-curl-7092-9051d0" Sep 15 11:57:14.833344 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:14.831251 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="236a4fb3-568e-418f-9dc4-75241f439daf" containerName="test-curl-7092-9051d0" Sep 15 11:57:14.833344 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:14.831329 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="236a4fb3-568e-418f-9dc4-75241f439daf" containerName="test-curl-7092-9051d0" Sep 15 11:57:14.834272 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:14.834252 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-d1e2fb" Sep 15 11:57:14.836856 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:14.836835 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dg4tz\"" Sep 15 11:57:14.840238 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:14.840219 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-d1e2fb"] Sep 15 11:57:14.935312 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:14.935275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr4dz\" (UniqueName: \"kubernetes.io/projected/a11a4207-7671-4686-9489-042d36a594f6-kube-api-access-pr4dz\") pod \"test-curl-7095-d1e2fb\" (UID: \"a11a4207-7671-4686-9489-042d36a594f6\") " pod="opendatahub/test-curl-7095-d1e2fb" Sep 15 11:57:15.036315 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:15.036279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr4dz\" (UniqueName: \"kubernetes.io/projected/a11a4207-7671-4686-9489-042d36a594f6-kube-api-access-pr4dz\") pod \"test-curl-7095-d1e2fb\" (UID: \"a11a4207-7671-4686-9489-042d36a594f6\") " pod="opendatahub/test-curl-7095-d1e2fb" Sep 15 11:57:15.044783 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:15.044748 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr4dz\" (UniqueName: \"kubernetes.io/projected/a11a4207-7671-4686-9489-042d36a594f6-kube-api-access-pr4dz\") pod \"test-curl-7095-d1e2fb\" (UID: \"a11a4207-7671-4686-9489-042d36a594f6\") " pod="opendatahub/test-curl-7095-d1e2fb" Sep 15 11:57:15.143955 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:15.143870 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-d1e2fb" Sep 15 11:57:15.415893 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:15.415869 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-d1e2fb"] Sep 15 11:57:15.421705 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:57:15.421670 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda11a4207_7671_4686_9489_042d36a594f6.slice/crio-7f05e2b99ca52078927148f2d775a0cb8b891df804aa8c5c5475dc820e50ea6d WatchSource:0}: Error finding container 7f05e2b99ca52078927148f2d775a0cb8b891df804aa8c5c5475dc820e50ea6d: Status 404 returned error can't find the container with id 7f05e2b99ca52078927148f2d775a0cb8b891df804aa8c5c5475dc820e50ea6d Sep 15 11:57:15.761801 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:15.761766 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-d1e2fb" event={"ID":"a11a4207-7671-4686-9489-042d36a594f6","Type":"ContainerStarted","Data":"e75eac36832c1750a83b5bfd99efe6adfc0ed034791b8bd1496639cce2c9ac33"} Sep 15 11:57:15.761998 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:15.761810 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-d1e2fb" event={"ID":"a11a4207-7671-4686-9489-042d36a594f6","Type":"ContainerStarted","Data":"7f05e2b99ca52078927148f2d775a0cb8b891df804aa8c5c5475dc820e50ea6d"} Sep 15 11:57:15.779370 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:15.779324 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7095-d1e2fb" podStartSLOduration=1.779309789 podStartE2EDuration="1.779309789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:57:15.777544064 +0000 UTC m=+1223.007470600" watchObservedRunningTime="2026-09-15 11:57:15.779309789 +0000 UTC m=+1223.009236357" Sep 15 11:57:16.377456 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:16.377416 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-d1e2fb"] Sep 15 11:57:16.380989 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:16.380956 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-d1e2fb"] Sep 15 11:57:16.766194 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:57:16.766156 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7095-d1e2fb" container="test-curl-7095-d1e2fb" Sep 15 11:57:17.770118 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:17.770081 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7095-d1e2fb" podUID="a11a4207-7671-4686-9489-042d36a594f6" containerName="test-curl-7095-d1e2fb" containerID="cri-o://e75eac36832c1750a83b5bfd99efe6adfc0ed034791b8bd1496639cce2c9ac33" gracePeriod=2 Sep 15 11:57:19.571231 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.571194 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp"] Sep 15 11:57:19.571606 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.571592 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a11a4207-7671-4686-9489-042d36a594f6" containerName="test-curl-7095-d1e2fb" Sep 15 11:57:19.571606 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.571607 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a11a4207-7671-4686-9489-042d36a594f6" containerName="test-curl-7095-d1e2fb" Sep 15 11:57:19.571686 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.571676 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a11a4207-7671-4686-9489-042d36a594f6" containerName="test-curl-7095-d1e2fb" Sep 15 11:57:19.575047 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.574999 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.578965 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.578926 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cc429a8e\"/\"e2e-lifecycle-model-cc429a8e-kserve-self-signed-certs\"" Sep 15 11:57:19.578965 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.578946 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cc429a8e\"/\"openshift-service-ca.crt\"" Sep 15 11:57:19.578965 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.578929 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cc429a8e\"/\"kube-root-ca.crt\"" Sep 15 11:57:19.579224 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.578934 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cc429a8e\"/\"default-dockercfg-7h92v\"" Sep 15 11:57:19.584904 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.584881 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp"] Sep 15 11:57:19.682611 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.682578 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-model-cache\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.682775 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.682620 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-home\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.682775 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.682689 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c9a2cb5-a387-4d6a-89af-539280bea35f-tls-certs\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.682860 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.682801 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-tmp-dir\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.682860 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.682825 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-dshm\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.682860 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.682852 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86c7x\" (UniqueName: \"kubernetes.io/projected/2c9a2cb5-a387-4d6a-89af-539280bea35f-kube-api-access-86c7x\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.783897 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.783854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-tmp-dir\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.783897 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.783887 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-dshm\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.783995 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.783907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86c7x\" (UniqueName: \"kubernetes.io/projected/2c9a2cb5-a387-4d6a-89af-539280bea35f-kube-api-access-86c7x\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.783995 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.783949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-model-cache\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.783995 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.783982 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-home\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.784178 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.784050 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c9a2cb5-a387-4d6a-89af-539280bea35f-tls-certs\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.784346 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.784303 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-tmp-dir\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.784573 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.784546 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-model-cache\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.784682 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.784578 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-home\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.786328 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.786306 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-dshm\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.786449 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.786428 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c9a2cb5-a387-4d6a-89af-539280bea35f-tls-certs\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.793289 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.793270 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86c7x\" (UniqueName: \"kubernetes.io/projected/2c9a2cb5-a387-4d6a-89af-539280bea35f-kube-api-access-86c7x\") pod \"e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.885333 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.885303 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:19.950259 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:19.950228 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-d1e2fb" Sep 15 11:57:20.086972 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:20.086866 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pr4dz\" (UniqueName: \"kubernetes.io/projected/a11a4207-7671-4686-9489-042d36a594f6-kube-api-access-pr4dz\") pod \"a11a4207-7671-4686-9489-042d36a594f6\" (UID: \"a11a4207-7671-4686-9489-042d36a594f6\") " Sep 15 11:57:20.088971 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:20.088948 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11a4207-7671-4686-9489-042d36a594f6-kube-api-access-pr4dz" (OuterVolumeSpecName: "kube-api-access-pr4dz") pod "a11a4207-7671-4686-9489-042d36a594f6" (UID: "a11a4207-7671-4686-9489-042d36a594f6"). InnerVolumeSpecName "kube-api-access-pr4dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:57:20.174112 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:20.174059 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp"] Sep 15 11:57:20.176362 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:57:20.176316 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c9a2cb5_a387_4d6a_89af_539280bea35f.slice/crio-0516cd5f05364b210e5c6f902ce5c6af8bfe66908959752066b703f30a040c22 WatchSource:0}: Error finding container 0516cd5f05364b210e5c6f902ce5c6af8bfe66908959752066b703f30a040c22: Status 404 returned error can't find the container with id 0516cd5f05364b210e5c6f902ce5c6af8bfe66908959752066b703f30a040c22 Sep 15 11:57:20.187628 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:20.187606 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr4dz\" (UniqueName: \"kubernetes.io/projected/a11a4207-7671-4686-9489-042d36a594f6-kube-api-access-pr4dz\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:57:20.787613 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:20.787576 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" event={"ID":"2c9a2cb5-a387-4d6a-89af-539280bea35f","Type":"ContainerStarted","Data":"22e294d37b41218e86bffb7a21512d4151e8ca6aae7be0e09cdab24f6e522b9b"} Sep 15 11:57:20.787963 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:20.787631 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" event={"ID":"2c9a2cb5-a387-4d6a-89af-539280bea35f","Type":"ContainerStarted","Data":"0516cd5f05364b210e5c6f902ce5c6af8bfe66908959752066b703f30a040c22"} Sep 15 11:57:20.788855 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:20.788830 2573 generic.go:356] "Generic (PLEG): container finished" podID="a11a4207-7671-4686-9489-042d36a594f6" containerID="e75eac36832c1750a83b5bfd99efe6adfc0ed034791b8bd1496639cce2c9ac33" exitCode=137 Sep 15 11:57:20.788956 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:20.788873 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-d1e2fb" Sep 15 11:57:20.788997 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:20.788931 2573 scope.go:117] "RemoveContainer" containerID="e75eac36832c1750a83b5bfd99efe6adfc0ed034791b8bd1496639cce2c9ac33" Sep 15 11:57:20.798267 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:20.798251 2573 scope.go:117] "RemoveContainer" containerID="e75eac36832c1750a83b5bfd99efe6adfc0ed034791b8bd1496639cce2c9ac33" Sep 15 11:57:20.798539 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:57:20.798520 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75eac36832c1750a83b5bfd99efe6adfc0ed034791b8bd1496639cce2c9ac33\": container with ID starting with e75eac36832c1750a83b5bfd99efe6adfc0ed034791b8bd1496639cce2c9ac33 not found: ID does not exist" containerID="e75eac36832c1750a83b5bfd99efe6adfc0ed034791b8bd1496639cce2c9ac33" Sep 15 11:57:20.798602 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:20.798547 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75eac36832c1750a83b5bfd99efe6adfc0ed034791b8bd1496639cce2c9ac33"} err="failed to get container status \"e75eac36832c1750a83b5bfd99efe6adfc0ed034791b8bd1496639cce2c9ac33\": rpc error: code = NotFound desc = could not find container \"e75eac36832c1750a83b5bfd99efe6adfc0ed034791b8bd1496639cce2c9ac33\": container with ID starting with e75eac36832c1750a83b5bfd99efe6adfc0ed034791b8bd1496639cce2c9ac33 not found: ID does not exist" Sep 15 11:57:20.806814 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:20.806768 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" podStartSLOduration=1.8067522409999999 podStartE2EDuration="1.806752241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:57:20.80559728 +0000 UTC m=+1228.035523815" watchObservedRunningTime="2026-09-15 11:57:20.806752241 +0000 UTC m=+1228.036678780" Sep 15 11:57:20.807473 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:57:20.807445 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7095-d1e2fb\" is forbidden: User \"system:node:ip-10-0-128-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-189.ec2.internal' and this object" podUID="a11a4207-7671-4686-9489-042d36a594f6" pod="opendatahub/test-curl-7095-d1e2fb" Sep 15 11:57:21.235083 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:21.234998 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a11a4207-7671-4686-9489-042d36a594f6" path="/var/lib/kubelet/pods/a11a4207-7671-4686-9489-042d36a594f6/volumes" Sep 15 11:57:21.580919 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:21.580884 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd"] Sep 15 11:57:29.885668 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:29.885633 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:29.886105 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:29.885684 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:29.898127 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:29.898106 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:30.850173 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:30.850140 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:57:36.599388 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.599341 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" podUID="b19f030d-447c-4856-abb2-835a861aaa15" containerName="main" containerID="cri-o://50294947b712265af0dbef4efc218add2c28cf8aaaa9c50c02a16f40137ab518" gracePeriod=45 Sep 15 11:57:36.843759 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.843735 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:57:36.867707 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.867624 2573 generic.go:356] "Generic (PLEG): container finished" podID="b19f030d-447c-4856-abb2-835a861aaa15" containerID="50294947b712265af0dbef4efc218add2c28cf8aaaa9c50c02a16f40137ab518" exitCode=0 Sep 15 11:57:36.867707 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.867667 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" Sep 15 11:57:36.867902 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.867706 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" event={"ID":"b19f030d-447c-4856-abb2-835a861aaa15","Type":"ContainerDied","Data":"50294947b712265af0dbef4efc218add2c28cf8aaaa9c50c02a16f40137ab518"} Sep 15 11:57:36.867902 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.867742 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd" event={"ID":"b19f030d-447c-4856-abb2-835a861aaa15","Type":"ContainerDied","Data":"ab45ec2f1f249b865b44812085e82e4a6d274868ed3b94ac6d1634a1952bc7f0"} Sep 15 11:57:36.867902 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.867763 2573 scope.go:117] "RemoveContainer" containerID="50294947b712265af0dbef4efc218add2c28cf8aaaa9c50c02a16f40137ab518" Sep 15 11:57:36.876592 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.876566 2573 scope.go:117] "RemoveContainer" containerID="50294947b712265af0dbef4efc218add2c28cf8aaaa9c50c02a16f40137ab518" Sep 15 11:57:36.876882 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:57:36.876864 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50294947b712265af0dbef4efc218add2c28cf8aaaa9c50c02a16f40137ab518\": container with ID starting with 50294947b712265af0dbef4efc218add2c28cf8aaaa9c50c02a16f40137ab518 not found: ID does not exist" containerID="50294947b712265af0dbef4efc218add2c28cf8aaaa9c50c02a16f40137ab518" Sep 15 11:57:36.876950 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.876890 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50294947b712265af0dbef4efc218add2c28cf8aaaa9c50c02a16f40137ab518"} err="failed to get container status \"50294947b712265af0dbef4efc218add2c28cf8aaaa9c50c02a16f40137ab518\": rpc error: code = NotFound desc = could not find container \"50294947b712265af0dbef4efc218add2c28cf8aaaa9c50c02a16f40137ab518\": container with ID starting with 50294947b712265af0dbef4efc218add2c28cf8aaaa9c50c02a16f40137ab518 not found: ID does not exist" Sep 15 11:57:36.931921 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.931896 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-dshm\") pod \"b19f030d-447c-4856-abb2-835a861aaa15\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " Sep 15 11:57:36.932060 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.931933 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-model-cache\") pod \"b19f030d-447c-4856-abb2-835a861aaa15\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " Sep 15 11:57:36.932060 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.931982 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19f030d-447c-4856-abb2-835a861aaa15-tls-certs\") pod \"b19f030d-447c-4856-abb2-835a861aaa15\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " Sep 15 11:57:36.932060 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.932000 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-tmp-dir\") pod \"b19f030d-447c-4856-abb2-835a861aaa15\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " Sep 15 11:57:36.932060 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.932053 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-home\") pod \"b19f030d-447c-4856-abb2-835a861aaa15\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " Sep 15 11:57:36.932291 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.932128 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxwh9\" (UniqueName: \"kubernetes.io/projected/b19f030d-447c-4856-abb2-835a861aaa15-kube-api-access-gxwh9\") pod \"b19f030d-447c-4856-abb2-835a861aaa15\" (UID: \"b19f030d-447c-4856-abb2-835a861aaa15\") " Sep 15 11:57:36.932291 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.932232 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b19f030d-447c-4856-abb2-835a861aaa15" (UID: "b19f030d-447c-4856-abb2-835a861aaa15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:57:36.932291 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.932258 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-model-cache" (OuterVolumeSpecName: "model-cache") pod "b19f030d-447c-4856-abb2-835a861aaa15" (UID: "b19f030d-447c-4856-abb2-835a861aaa15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:57:36.932458 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.932330 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-home" (OuterVolumeSpecName: "home") pod "b19f030d-447c-4856-abb2-835a861aaa15" (UID: "b19f030d-447c-4856-abb2-835a861aaa15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:57:36.932511 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.932465 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-model-cache\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:57:36.932511 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.932483 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-tmp-dir\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:57:36.932511 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.932492 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-home\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:57:36.934228 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.934203 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b19f030d-447c-4856-abb2-835a861aaa15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b19f030d-447c-4856-abb2-835a861aaa15" (UID: "b19f030d-447c-4856-abb2-835a861aaa15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:57:36.934228 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.934215 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19f030d-447c-4856-abb2-835a861aaa15-kube-api-access-gxwh9" (OuterVolumeSpecName: "kube-api-access-gxwh9") pod "b19f030d-447c-4856-abb2-835a861aaa15" (UID: "b19f030d-447c-4856-abb2-835a861aaa15"). InnerVolumeSpecName "kube-api-access-gxwh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:57:36.934358 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:36.934235 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-dshm" (OuterVolumeSpecName: "dshm") pod "b19f030d-447c-4856-abb2-835a861aaa15" (UID: "b19f030d-447c-4856-abb2-835a861aaa15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:57:37.032849 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:37.032819 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxwh9\" (UniqueName: \"kubernetes.io/projected/b19f030d-447c-4856-abb2-835a861aaa15-kube-api-access-gxwh9\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:57:37.032849 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:37.032844 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19f030d-447c-4856-abb2-835a861aaa15-dshm\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:57:37.032849 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:37.032855 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19f030d-447c-4856-abb2-835a861aaa15-tls-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:57:37.188863 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:37.188836 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd"] Sep 15 11:57:37.192663 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:37.192642 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-42b26e/auth-test-model-42b26e-kserve-75c65d475-nlxgd"] Sep 15 11:57:37.234712 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:57:37.234675 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b19f030d-447c-4856-abb2-835a861aaa15" path="/var/lib/kubelet/pods/b19f030d-447c-4856-abb2-835a861aaa15/volumes" Sep 15 11:58:18.258788 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:18.258747 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp"] Sep 15 11:58:26.300361 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.300326 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr"] Sep 15 11:58:26.300818 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.300712 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b19f030d-447c-4856-abb2-835a861aaa15" containerName="main" Sep 15 11:58:26.300818 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.300725 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b19f030d-447c-4856-abb2-835a861aaa15" containerName="main" Sep 15 11:58:26.300818 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.300807 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b19f030d-447c-4856-abb2-835a861aaa15" containerName="main" Sep 15 11:58:26.303909 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.303884 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.307269 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.307248 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-7b7b3c\"/\"kube-root-ca.crt\"" Sep 15 11:58:26.307373 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.307351 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-7b7b3c\"/\"openshift-service-ca.crt\"" Sep 15 11:58:26.307435 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.307371 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-7b7b3c\"/\"default-dockercfg-h9z4x\"" Sep 15 11:58:26.307435 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.307389 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-7b7b3c\"/\"sub-test-model-7b7b3c-kserve-self-signed-certs\"" Sep 15 11:58:26.312989 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.312966 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr"] Sep 15 11:58:26.452799 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.452766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-dshm\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.452799 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.452803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-home\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.453013 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.452832 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-tmp-dir\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.453013 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.452850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63665c9-9691-4fb9-9b67-eecaaf01e900-tls-certs\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.453013 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.452896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4stlv\" (UniqueName: \"kubernetes.io/projected/f63665c9-9691-4fb9-9b67-eecaaf01e900-kube-api-access-4stlv\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.453013 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.452920 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-model-cache\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.553615 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.553528 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-model-cache\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.553615 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.553612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-dshm\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.553846 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.553634 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-home\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.553846 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.553662 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-tmp-dir\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.553846 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.553680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63665c9-9691-4fb9-9b67-eecaaf01e900-tls-certs\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.553846 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.553714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4stlv\" (UniqueName: \"kubernetes.io/projected/f63665c9-9691-4fb9-9b67-eecaaf01e900-kube-api-access-4stlv\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.554085 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.553993 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-model-cache\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.554284 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.554255 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-tmp-dir\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.554414 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.554306 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-home\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.556228 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.556198 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-dshm\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.556330 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.556225 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63665c9-9691-4fb9-9b67-eecaaf01e900-tls-certs\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.562086 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.562056 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4stlv\" (UniqueName: \"kubernetes.io/projected/f63665c9-9691-4fb9-9b67-eecaaf01e900-kube-api-access-4stlv\") pod \"sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.613970 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.613944 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:26.883896 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:26.883871 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr"] Sep 15 11:58:26.885984 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:58:26.885958 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf63665c9_9691_4fb9_9b67_eecaaf01e900.slice/crio-2f1cded031982e3e4b9d6d96df19f08bddfcffa4f3c7bb3f437b19674c365142 WatchSource:0}: Error finding container 2f1cded031982e3e4b9d6d96df19f08bddfcffa4f3c7bb3f437b19674c365142: Status 404 returned error can't find the container with id 2f1cded031982e3e4b9d6d96df19f08bddfcffa4f3c7bb3f437b19674c365142 Sep 15 11:58:27.111477 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:27.111444 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" event={"ID":"f63665c9-9691-4fb9-9b67-eecaaf01e900","Type":"ContainerStarted","Data":"68a50a3569551775dda22be1444de479a7a301ec711150bae41f910d75fedcc6"} Sep 15 11:58:27.111636 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:27.111486 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" event={"ID":"f63665c9-9691-4fb9-9b67-eecaaf01e900","Type":"ContainerStarted","Data":"2f1cded031982e3e4b9d6d96df19f08bddfcffa4f3c7bb3f437b19674c365142"} Sep 15 11:58:27.133841 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:27.133752 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" podStartSLOduration=1.133735292 podStartE2EDuration="1.133735292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:58:27.131771133 +0000 UTC m=+1294.361697812" watchObservedRunningTime="2026-09-15 11:58:27.133735292 +0000 UTC m=+1294.363661827" Sep 15 11:58:33.278059 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.277998 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" podUID="2c9a2cb5-a387-4d6a-89af-539280bea35f" containerName="main" containerID="cri-o://22e294d37b41218e86bffb7a21512d4151e8ca6aae7be0e09cdab24f6e522b9b" gracePeriod=45 Sep 15 11:58:33.532869 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.532806 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:58:33.613768 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.613737 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-model-cache\") pod \"2c9a2cb5-a387-4d6a-89af-539280bea35f\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " Sep 15 11:58:33.613930 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.613778 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-tmp-dir\") pod \"2c9a2cb5-a387-4d6a-89af-539280bea35f\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " Sep 15 11:58:33.613930 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.613809 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-dshm\") pod \"2c9a2cb5-a387-4d6a-89af-539280bea35f\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " Sep 15 11:58:33.613930 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.613875 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c9a2cb5-a387-4d6a-89af-539280bea35f-tls-certs\") pod \"2c9a2cb5-a387-4d6a-89af-539280bea35f\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " Sep 15 11:58:33.613930 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.613911 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86c7x\" (UniqueName: \"kubernetes.io/projected/2c9a2cb5-a387-4d6a-89af-539280bea35f-kube-api-access-86c7x\") pod \"2c9a2cb5-a387-4d6a-89af-539280bea35f\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " Sep 15 11:58:33.614169 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.613950 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-home\") pod \"2c9a2cb5-a387-4d6a-89af-539280bea35f\" (UID: \"2c9a2cb5-a387-4d6a-89af-539280bea35f\") " Sep 15 11:58:33.614169 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.614041 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c9a2cb5-a387-4d6a-89af-539280bea35f" (UID: "2c9a2cb5-a387-4d6a-89af-539280bea35f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:58:33.614169 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.614068 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c9a2cb5-a387-4d6a-89af-539280bea35f" (UID: "2c9a2cb5-a387-4d6a-89af-539280bea35f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:58:33.614335 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.614259 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-model-cache\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:58:33.614335 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.614258 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-home" (OuterVolumeSpecName: "home") pod "2c9a2cb5-a387-4d6a-89af-539280bea35f" (UID: "2c9a2cb5-a387-4d6a-89af-539280bea35f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:58:33.614335 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.614273 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-tmp-dir\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:58:33.616103 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.616073 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-dshm" (OuterVolumeSpecName: "dshm") pod "2c9a2cb5-a387-4d6a-89af-539280bea35f" (UID: "2c9a2cb5-a387-4d6a-89af-539280bea35f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:58:33.616209 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.616165 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9a2cb5-a387-4d6a-89af-539280bea35f-kube-api-access-86c7x" (OuterVolumeSpecName: "kube-api-access-86c7x") pod "2c9a2cb5-a387-4d6a-89af-539280bea35f" (UID: "2c9a2cb5-a387-4d6a-89af-539280bea35f"). InnerVolumeSpecName "kube-api-access-86c7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:58:33.616251 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.616231 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9a2cb5-a387-4d6a-89af-539280bea35f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c9a2cb5-a387-4d6a-89af-539280bea35f" (UID: "2c9a2cb5-a387-4d6a-89af-539280bea35f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:58:33.715683 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.715633 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c9a2cb5-a387-4d6a-89af-539280bea35f-tls-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:58:33.715683 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.715678 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86c7x\" (UniqueName: \"kubernetes.io/projected/2c9a2cb5-a387-4d6a-89af-539280bea35f-kube-api-access-86c7x\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:58:33.715683 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.715692 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-home\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:58:33.715914 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:33.715705 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c9a2cb5-a387-4d6a-89af-539280bea35f-dshm\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:58:34.147985 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:34.147948 2573 generic.go:356] "Generic (PLEG): container finished" podID="2c9a2cb5-a387-4d6a-89af-539280bea35f" containerID="22e294d37b41218e86bffb7a21512d4151e8ca6aae7be0e09cdab24f6e522b9b" exitCode=0 Sep 15 11:58:34.148193 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:34.147994 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" Sep 15 11:58:34.148193 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:34.148049 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" event={"ID":"2c9a2cb5-a387-4d6a-89af-539280bea35f","Type":"ContainerDied","Data":"22e294d37b41218e86bffb7a21512d4151e8ca6aae7be0e09cdab24f6e522b9b"} Sep 15 11:58:34.148193 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:34.148101 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp" event={"ID":"2c9a2cb5-a387-4d6a-89af-539280bea35f","Type":"ContainerDied","Data":"0516cd5f05364b210e5c6f902ce5c6af8bfe66908959752066b703f30a040c22"} Sep 15 11:58:34.148193 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:34.148125 2573 scope.go:117] "RemoveContainer" containerID="22e294d37b41218e86bffb7a21512d4151e8ca6aae7be0e09cdab24f6e522b9b" Sep 15 11:58:34.157072 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:34.157055 2573 scope.go:117] "RemoveContainer" containerID="22e294d37b41218e86bffb7a21512d4151e8ca6aae7be0e09cdab24f6e522b9b" Sep 15 11:58:34.157333 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:58:34.157314 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e294d37b41218e86bffb7a21512d4151e8ca6aae7be0e09cdab24f6e522b9b\": container with ID starting with 22e294d37b41218e86bffb7a21512d4151e8ca6aae7be0e09cdab24f6e522b9b not found: ID does not exist" containerID="22e294d37b41218e86bffb7a21512d4151e8ca6aae7be0e09cdab24f6e522b9b" Sep 15 11:58:34.157397 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:34.157345 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e294d37b41218e86bffb7a21512d4151e8ca6aae7be0e09cdab24f6e522b9b"} err="failed to get container status \"22e294d37b41218e86bffb7a21512d4151e8ca6aae7be0e09cdab24f6e522b9b\": rpc error: code = NotFound desc = could not find container \"22e294d37b41218e86bffb7a21512d4151e8ca6aae7be0e09cdab24f6e522b9b\": container with ID starting with 22e294d37b41218e86bffb7a21512d4151e8ca6aae7be0e09cdab24f6e522b9b not found: ID does not exist" Sep 15 11:58:34.171897 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:34.171875 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp"] Sep 15 11:58:34.176126 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:34.176097 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cc429a8e/e2e-lifecycle-model-cc429a8e-kserve-6f77dc64cf-9wtbp"] Sep 15 11:58:35.234965 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:35.234927 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c9a2cb5-a387-4d6a-89af-539280bea35f" path="/var/lib/kubelet/pods/2c9a2cb5-a387-4d6a-89af-539280bea35f/volumes" Sep 15 11:58:36.614844 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:36.614806 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:36.615262 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:36.614883 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:36.627013 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:36.626995 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:58:37.176075 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:58:37.176046 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:59:12.927091 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:12.927054 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-fd54e7"] Sep 15 11:59:12.927556 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:12.927412 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9a2cb5-a387-4d6a-89af-539280bea35f" containerName="main" Sep 15 11:59:12.927556 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:12.927424 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9a2cb5-a387-4d6a-89af-539280bea35f" containerName="main" Sep 15 11:59:12.927556 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:12.927488 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9a2cb5-a387-4d6a-89af-539280bea35f" containerName="main" Sep 15 11:59:12.931973 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:12.931952 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-fd54e7" Sep 15 11:59:12.934420 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:12.934393 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dg4tz\"" Sep 15 11:59:12.938552 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:12.938530 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-fd54e7"] Sep 15 11:59:13.044782 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:13.044752 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wh5v\" (UniqueName: \"kubernetes.io/projected/b5ca187e-1f7e-40c6-a9b3-248120278afb-kube-api-access-9wh5v\") pod \"test-curl-7095-fd54e7\" (UID: \"b5ca187e-1f7e-40c6-a9b3-248120278afb\") " pod="opendatahub/test-curl-7095-fd54e7" Sep 15 11:59:13.145586 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:13.145546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wh5v\" (UniqueName: \"kubernetes.io/projected/b5ca187e-1f7e-40c6-a9b3-248120278afb-kube-api-access-9wh5v\") pod \"test-curl-7095-fd54e7\" (UID: \"b5ca187e-1f7e-40c6-a9b3-248120278afb\") " pod="opendatahub/test-curl-7095-fd54e7" Sep 15 11:59:13.154434 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:13.154404 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wh5v\" (UniqueName: \"kubernetes.io/projected/b5ca187e-1f7e-40c6-a9b3-248120278afb-kube-api-access-9wh5v\") pod \"test-curl-7095-fd54e7\" (UID: \"b5ca187e-1f7e-40c6-a9b3-248120278afb\") " pod="opendatahub/test-curl-7095-fd54e7" Sep 15 11:59:13.242860 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:13.242777 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-fd54e7" Sep 15 11:59:13.509959 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:13.509934 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-fd54e7"] Sep 15 11:59:13.512426 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:59:13.512389 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5ca187e_1f7e_40c6_a9b3_248120278afb.slice/crio-c7dbb54c2706ce4258674aa0b39620b736485d37587af2668a9ecaae9dfc26a4 WatchSource:0}: Error finding container c7dbb54c2706ce4258674aa0b39620b736485d37587af2668a9ecaae9dfc26a4: Status 404 returned error can't find the container with id c7dbb54c2706ce4258674aa0b39620b736485d37587af2668a9ecaae9dfc26a4 Sep 15 11:59:14.339164 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:14.339130 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-fd54e7" event={"ID":"b5ca187e-1f7e-40c6-a9b3-248120278afb","Type":"ContainerStarted","Data":"ad5cf37d31e9c0768f650ce5fcf94b08f395e2799d7c8779d02b8174a4ef923e"} Sep 15 11:59:14.339545 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:14.339172 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-fd54e7" event={"ID":"b5ca187e-1f7e-40c6-a9b3-248120278afb","Type":"ContainerStarted","Data":"c7dbb54c2706ce4258674aa0b39620b736485d37587af2668a9ecaae9dfc26a4"} Sep 15 11:59:14.352851 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:14.352802 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7095-fd54e7" podStartSLOduration=2.352785851 podStartE2EDuration="2.352785851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:14.352621213 +0000 UTC m=+1341.582547777" watchObservedRunningTime="2026-09-15 11:59:14.352785851 +0000 UTC m=+1341.582712388" Sep 15 11:59:14.949772 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:14.949738 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-fd54e7"] Sep 15 11:59:14.953764 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:14.953734 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-fd54e7"] Sep 15 11:59:15.070930 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:15.070894 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-608cde"] Sep 15 11:59:15.071310 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:15.071296 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5ca187e-1f7e-40c6-a9b3-248120278afb" containerName="test-curl-7095-fd54e7" Sep 15 11:59:15.071310 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:15.071311 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5ca187e-1f7e-40c6-a9b3-248120278afb" containerName="test-curl-7095-fd54e7" Sep 15 11:59:15.071419 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:15.071387 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5ca187e-1f7e-40c6-a9b3-248120278afb" containerName="test-curl-7095-fd54e7" Sep 15 11:59:15.074395 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:15.074377 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-608cde" Sep 15 11:59:15.082228 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:15.082199 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-608cde"] Sep 15 11:59:15.165128 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:15.165095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7smd\" (UniqueName: \"kubernetes.io/projected/e35f9482-28c5-4c92-bf23-5e0827cb7121-kube-api-access-l7smd\") pod \"test-curl-7095-608cde\" (UID: \"e35f9482-28c5-4c92-bf23-5e0827cb7121\") " pod="opendatahub/test-curl-7095-608cde" Sep 15 11:59:15.266296 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:15.266262 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7smd\" (UniqueName: \"kubernetes.io/projected/e35f9482-28c5-4c92-bf23-5e0827cb7121-kube-api-access-l7smd\") pod \"test-curl-7095-608cde\" (UID: \"e35f9482-28c5-4c92-bf23-5e0827cb7121\") " pod="opendatahub/test-curl-7095-608cde" Sep 15 11:59:15.274485 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:15.274460 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7smd\" (UniqueName: \"kubernetes.io/projected/e35f9482-28c5-4c92-bf23-5e0827cb7121-kube-api-access-l7smd\") pod \"test-curl-7095-608cde\" (UID: \"e35f9482-28c5-4c92-bf23-5e0827cb7121\") " pod="opendatahub/test-curl-7095-608cde" Sep 15 11:59:15.343879 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:59:15.343809 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7095-fd54e7" container="test-curl-7095-fd54e7" Sep 15 11:59:15.385040 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:15.384997 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-608cde" Sep 15 11:59:15.653924 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:15.653848 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-608cde"] Sep 15 11:59:15.656676 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:59:15.656633 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode35f9482_28c5_4c92_bf23_5e0827cb7121.slice/crio-2b71dc20ef505a3ee2d7961b9dfdab722611b0ccae8e36ea3a035b9a8263f322 WatchSource:0}: Error finding container 2b71dc20ef505a3ee2d7961b9dfdab722611b0ccae8e36ea3a035b9a8263f322: Status 404 returned error can't find the container with id 2b71dc20ef505a3ee2d7961b9dfdab722611b0ccae8e36ea3a035b9a8263f322 Sep 15 11:59:16.349898 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:16.349858 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-608cde" event={"ID":"e35f9482-28c5-4c92-bf23-5e0827cb7121","Type":"ContainerStarted","Data":"cd4df91f7921def2b48f65b4c58f699be02b9536fc613e3f1f0b6e9aff0bc02a"} Sep 15 11:59:16.349898 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:16.349902 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-608cde" event={"ID":"e35f9482-28c5-4c92-bf23-5e0827cb7121","Type":"ContainerStarted","Data":"2b71dc20ef505a3ee2d7961b9dfdab722611b0ccae8e36ea3a035b9a8263f322"} Sep 15 11:59:16.350398 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:16.349990 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7095-fd54e7" podUID="b5ca187e-1f7e-40c6-a9b3-248120278afb" containerName="test-curl-7095-fd54e7" containerID="cri-o://ad5cf37d31e9c0768f650ce5fcf94b08f395e2799d7c8779d02b8174a4ef923e" gracePeriod=2 Sep 15 11:59:16.363320 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:16.363259 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7095-608cde" podStartSLOduration=1.363242404 podStartE2EDuration="1.363242404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:16.362980963 +0000 UTC m=+1343.592907500" watchObservedRunningTime="2026-09-15 11:59:16.363242404 +0000 UTC m=+1343.593168939" Sep 15 11:59:16.944942 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:16.944910 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-608cde"] Sep 15 11:59:16.948650 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:16.948622 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-608cde"] Sep 15 11:59:17.354520 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:59:17.354484 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7095-608cde" container="test-curl-7095-608cde" Sep 15 11:59:18.360165 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:18.360126 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7095-608cde" podUID="e35f9482-28c5-4c92-bf23-5e0827cb7121" containerName="test-curl-7095-608cde" containerID="cri-o://cd4df91f7921def2b48f65b4c58f699be02b9536fc613e3f1f0b6e9aff0bc02a" gracePeriod=2 Sep 15 11:59:18.529096 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:18.529072 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-fd54e7" Sep 15 11:59:18.696136 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:18.696059 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wh5v\" (UniqueName: \"kubernetes.io/projected/b5ca187e-1f7e-40c6-a9b3-248120278afb-kube-api-access-9wh5v\") pod \"b5ca187e-1f7e-40c6-a9b3-248120278afb\" (UID: \"b5ca187e-1f7e-40c6-a9b3-248120278afb\") " Sep 15 11:59:18.698076 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:18.698047 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ca187e-1f7e-40c6-a9b3-248120278afb-kube-api-access-9wh5v" (OuterVolumeSpecName: "kube-api-access-9wh5v") pod "b5ca187e-1f7e-40c6-a9b3-248120278afb" (UID: "b5ca187e-1f7e-40c6-a9b3-248120278afb"). InnerVolumeSpecName "kube-api-access-9wh5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:59:18.797570 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:18.797539 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wh5v\" (UniqueName: \"kubernetes.io/projected/b5ca187e-1f7e-40c6-a9b3-248120278afb-kube-api-access-9wh5v\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:59:19.234905 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:19.234871 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5ca187e-1f7e-40c6-a9b3-248120278afb" path="/var/lib/kubelet/pods/b5ca187e-1f7e-40c6-a9b3-248120278afb/volumes" Sep 15 11:59:19.365722 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:19.365681 2573 generic.go:356] "Generic (PLEG): container finished" podID="b5ca187e-1f7e-40c6-a9b3-248120278afb" containerID="ad5cf37d31e9c0768f650ce5fcf94b08f395e2799d7c8779d02b8174a4ef923e" exitCode=137 Sep 15 11:59:19.365722 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:19.365724 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-fd54e7" Sep 15 11:59:19.366191 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:19.365780 2573 scope.go:117] "RemoveContainer" containerID="ad5cf37d31e9c0768f650ce5fcf94b08f395e2799d7c8779d02b8174a4ef923e" Sep 15 11:59:19.374218 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:19.374197 2573 scope.go:117] "RemoveContainer" containerID="ad5cf37d31e9c0768f650ce5fcf94b08f395e2799d7c8779d02b8174a4ef923e" Sep 15 11:59:19.374524 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:59:19.374503 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5cf37d31e9c0768f650ce5fcf94b08f395e2799d7c8779d02b8174a4ef923e\": container with ID starting with ad5cf37d31e9c0768f650ce5fcf94b08f395e2799d7c8779d02b8174a4ef923e not found: ID does not exist" containerID="ad5cf37d31e9c0768f650ce5fcf94b08f395e2799d7c8779d02b8174a4ef923e" Sep 15 11:59:19.374615 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:19.374533 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5cf37d31e9c0768f650ce5fcf94b08f395e2799d7c8779d02b8174a4ef923e"} err="failed to get container status \"ad5cf37d31e9c0768f650ce5fcf94b08f395e2799d7c8779d02b8174a4ef923e\": rpc error: code = NotFound desc = could not find container \"ad5cf37d31e9c0768f650ce5fcf94b08f395e2799d7c8779d02b8174a4ef923e\": container with ID starting with ad5cf37d31e9c0768f650ce5fcf94b08f395e2799d7c8779d02b8174a4ef923e not found: ID does not exist" Sep 15 11:59:19.931270 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:19.931238 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr"] Sep 15 11:59:20.550451 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:20.550423 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-608cde" Sep 15 11:59:20.711784 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:20.711700 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7smd\" (UniqueName: \"kubernetes.io/projected/e35f9482-28c5-4c92-bf23-5e0827cb7121-kube-api-access-l7smd\") pod \"e35f9482-28c5-4c92-bf23-5e0827cb7121\" (UID: \"e35f9482-28c5-4c92-bf23-5e0827cb7121\") " Sep 15 11:59:20.713732 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:20.713702 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35f9482-28c5-4c92-bf23-5e0827cb7121-kube-api-access-l7smd" (OuterVolumeSpecName: "kube-api-access-l7smd") pod "e35f9482-28c5-4c92-bf23-5e0827cb7121" (UID: "e35f9482-28c5-4c92-bf23-5e0827cb7121"). InnerVolumeSpecName "kube-api-access-l7smd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:59:20.812769 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:20.812732 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7smd\" (UniqueName: \"kubernetes.io/projected/e35f9482-28c5-4c92-bf23-5e0827cb7121-kube-api-access-l7smd\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:59:21.234721 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:21.234681 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e35f9482-28c5-4c92-bf23-5e0827cb7121" path="/var/lib/kubelet/pods/e35f9482-28c5-4c92-bf23-5e0827cb7121/volumes" Sep 15 11:59:21.377748 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:21.377716 2573 generic.go:356] "Generic (PLEG): container finished" podID="e35f9482-28c5-4c92-bf23-5e0827cb7121" containerID="cd4df91f7921def2b48f65b4c58f699be02b9536fc613e3f1f0b6e9aff0bc02a" exitCode=137 Sep 15 11:59:21.377939 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:21.377755 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-608cde" Sep 15 11:59:21.377939 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:21.377788 2573 scope.go:117] "RemoveContainer" containerID="cd4df91f7921def2b48f65b4c58f699be02b9536fc613e3f1f0b6e9aff0bc02a" Sep 15 11:59:21.386226 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:21.386206 2573 scope.go:117] "RemoveContainer" containerID="cd4df91f7921def2b48f65b4c58f699be02b9536fc613e3f1f0b6e9aff0bc02a" Sep 15 11:59:21.386488 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:59:21.386463 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4df91f7921def2b48f65b4c58f699be02b9536fc613e3f1f0b6e9aff0bc02a\": container with ID starting with cd4df91f7921def2b48f65b4c58f699be02b9536fc613e3f1f0b6e9aff0bc02a not found: ID does not exist" containerID="cd4df91f7921def2b48f65b4c58f699be02b9536fc613e3f1f0b6e9aff0bc02a" Sep 15 11:59:21.386598 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:21.386493 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4df91f7921def2b48f65b4c58f699be02b9536fc613e3f1f0b6e9aff0bc02a"} err="failed to get container status \"cd4df91f7921def2b48f65b4c58f699be02b9536fc613e3f1f0b6e9aff0bc02a\": rpc error: code = NotFound desc = could not find container \"cd4df91f7921def2b48f65b4c58f699be02b9536fc613e3f1f0b6e9aff0bc02a\": container with ID starting with cd4df91f7921def2b48f65b4c58f699be02b9536fc613e3f1f0b6e9aff0bc02a not found: ID does not exist" Sep 15 11:59:34.950260 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:34.950197 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" podUID="f63665c9-9691-4fb9-9b67-eecaaf01e900" containerName="main" containerID="cri-o://68a50a3569551775dda22be1444de479a7a301ec711150bae41f910d75fedcc6" gracePeriod=45 Sep 15 11:59:35.196739 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.196712 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:59:35.329809 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.329779 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63665c9-9691-4fb9-9b67-eecaaf01e900-tls-certs\") pod \"f63665c9-9691-4fb9-9b67-eecaaf01e900\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " Sep 15 11:59:35.329962 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.329835 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4stlv\" (UniqueName: \"kubernetes.io/projected/f63665c9-9691-4fb9-9b67-eecaaf01e900-kube-api-access-4stlv\") pod \"f63665c9-9691-4fb9-9b67-eecaaf01e900\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " Sep 15 11:59:35.329962 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.329866 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-model-cache\") pod \"f63665c9-9691-4fb9-9b67-eecaaf01e900\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " Sep 15 11:59:35.329962 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.329882 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-tmp-dir\") pod \"f63665c9-9691-4fb9-9b67-eecaaf01e900\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " Sep 15 11:59:35.329962 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.329941 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-dshm\") pod \"f63665c9-9691-4fb9-9b67-eecaaf01e900\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " Sep 15 11:59:35.330238 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.329972 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-home\") pod \"f63665c9-9691-4fb9-9b67-eecaaf01e900\" (UID: \"f63665c9-9691-4fb9-9b67-eecaaf01e900\") " Sep 15 11:59:35.330238 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.330195 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-model-cache" (OuterVolumeSpecName: "model-cache") pod "f63665c9-9691-4fb9-9b67-eecaaf01e900" (UID: "f63665c9-9691-4fb9-9b67-eecaaf01e900"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:59:35.330238 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.330208 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f63665c9-9691-4fb9-9b67-eecaaf01e900" (UID: "f63665c9-9691-4fb9-9b67-eecaaf01e900"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:59:35.330393 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.330306 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-home" (OuterVolumeSpecName: "home") pod "f63665c9-9691-4fb9-9b67-eecaaf01e900" (UID: "f63665c9-9691-4fb9-9b67-eecaaf01e900"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:59:35.330543 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.330461 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-home\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:59:35.330543 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.330481 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-model-cache\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:59:35.330543 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.330492 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-tmp-dir\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:59:35.332237 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.332206 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-dshm" (OuterVolumeSpecName: "dshm") pod "f63665c9-9691-4fb9-9b67-eecaaf01e900" (UID: "f63665c9-9691-4fb9-9b67-eecaaf01e900"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:59:35.332237 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.332217 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63665c9-9691-4fb9-9b67-eecaaf01e900-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f63665c9-9691-4fb9-9b67-eecaaf01e900" (UID: "f63665c9-9691-4fb9-9b67-eecaaf01e900"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:59:35.332362 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.332222 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63665c9-9691-4fb9-9b67-eecaaf01e900-kube-api-access-4stlv" (OuterVolumeSpecName: "kube-api-access-4stlv") pod "f63665c9-9691-4fb9-9b67-eecaaf01e900" (UID: "f63665c9-9691-4fb9-9b67-eecaaf01e900"). InnerVolumeSpecName "kube-api-access-4stlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:59:35.431755 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.431729 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f63665c9-9691-4fb9-9b67-eecaaf01e900-dshm\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:59:35.431755 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.431752 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63665c9-9691-4fb9-9b67-eecaaf01e900-tls-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:59:35.431898 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.431764 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4stlv\" (UniqueName: \"kubernetes.io/projected/f63665c9-9691-4fb9-9b67-eecaaf01e900-kube-api-access-4stlv\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 11:59:35.446889 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.446861 2573 generic.go:356] "Generic (PLEG): container finished" podID="f63665c9-9691-4fb9-9b67-eecaaf01e900" containerID="68a50a3569551775dda22be1444de479a7a301ec711150bae41f910d75fedcc6" exitCode=0 Sep 15 11:59:35.447001 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.446922 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" Sep 15 11:59:35.447001 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.446952 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" event={"ID":"f63665c9-9691-4fb9-9b67-eecaaf01e900","Type":"ContainerDied","Data":"68a50a3569551775dda22be1444de479a7a301ec711150bae41f910d75fedcc6"} Sep 15 11:59:35.447001 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.446999 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr" event={"ID":"f63665c9-9691-4fb9-9b67-eecaaf01e900","Type":"ContainerDied","Data":"2f1cded031982e3e4b9d6d96df19f08bddfcffa4f3c7bb3f437b19674c365142"} Sep 15 11:59:35.447145 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.447017 2573 scope.go:117] "RemoveContainer" containerID="68a50a3569551775dda22be1444de479a7a301ec711150bae41f910d75fedcc6" Sep 15 11:59:35.456098 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.456081 2573 scope.go:117] "RemoveContainer" containerID="68a50a3569551775dda22be1444de479a7a301ec711150bae41f910d75fedcc6" Sep 15 11:59:35.456372 ip-10-0-128-189 kubenswrapper[2573]: E0915 11:59:35.456356 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a50a3569551775dda22be1444de479a7a301ec711150bae41f910d75fedcc6\": container with ID starting with 68a50a3569551775dda22be1444de479a7a301ec711150bae41f910d75fedcc6 not found: ID does not exist" containerID="68a50a3569551775dda22be1444de479a7a301ec711150bae41f910d75fedcc6" Sep 15 11:59:35.456414 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.456378 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a50a3569551775dda22be1444de479a7a301ec711150bae41f910d75fedcc6"} err="failed to get container status \"68a50a3569551775dda22be1444de479a7a301ec711150bae41f910d75fedcc6\": rpc error: code = NotFound desc = could not find container \"68a50a3569551775dda22be1444de479a7a301ec711150bae41f910d75fedcc6\": container with ID starting with 68a50a3569551775dda22be1444de479a7a301ec711150bae41f910d75fedcc6 not found: ID does not exist" Sep 15 11:59:35.468408 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.468385 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr"] Sep 15 11:59:35.469995 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:35.469975 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7b7b3c/sub-test-model-7b7b3c-kserve-699fd68bb7-hpnxr"] Sep 15 11:59:37.235588 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:37.235547 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f63665c9-9691-4fb9-9b67-eecaaf01e900" path="/var/lib/kubelet/pods/f63665c9-9691-4fb9-9b67-eecaaf01e900/volumes" Sep 15 11:59:40.184271 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.184196 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c"] Sep 15 11:59:40.184614 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.184549 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63665c9-9691-4fb9-9b67-eecaaf01e900" containerName="main" Sep 15 11:59:40.184614 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.184560 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63665c9-9691-4fb9-9b67-eecaaf01e900" containerName="main" Sep 15 11:59:40.184614 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.184582 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e35f9482-28c5-4c92-bf23-5e0827cb7121" containerName="test-curl-7095-608cde" Sep 15 11:59:40.184614 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.184590 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e35f9482-28c5-4c92-bf23-5e0827cb7121" containerName="test-curl-7095-608cde" Sep 15 11:59:40.184751 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.184645 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e35f9482-28c5-4c92-bf23-5e0827cb7121" containerName="test-curl-7095-608cde" Sep 15 11:59:40.184751 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.184654 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63665c9-9691-4fb9-9b67-eecaaf01e900" containerName="main" Sep 15 11:59:40.188679 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.188662 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.192335 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.192302 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-42b26e\"/\"openshift-service-ca.crt\"" Sep 15 11:59:40.192335 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.192320 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-42b26e\"/\"kube-root-ca.crt\"" Sep 15 11:59:40.192335 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.192332 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-42b26e\"/\"rate-test-model-42b26e-kserve-self-signed-certs\"" Sep 15 11:59:40.192602 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.192301 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-42b26e\"/\"default-dockercfg-77nkl\"" Sep 15 11:59:40.196372 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.196351 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c"] Sep 15 11:59:40.271516 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.271480 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-dshm\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.271670 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.271624 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-home\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.271747 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.271672 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/830595fb-e30c-4db5-b0dc-356e37de36f8-tls-certs\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.271747 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.271702 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcc6v\" (UniqueName: \"kubernetes.io/projected/830595fb-e30c-4db5-b0dc-356e37de36f8-kube-api-access-hcc6v\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.271944 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.271920 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-model-cache\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.272059 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.271995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-tmp-dir\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.372607 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.372565 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-model-cache\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.372607 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.372612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-tmp-dir\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.372881 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.372639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-dshm\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.372881 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.372661 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-home\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.372881 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.372681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/830595fb-e30c-4db5-b0dc-356e37de36f8-tls-certs\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.372881 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.372697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcc6v\" (UniqueName: \"kubernetes.io/projected/830595fb-e30c-4db5-b0dc-356e37de36f8-kube-api-access-hcc6v\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.373125 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.373051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-model-cache\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.373189 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.373117 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-home\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.373254 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.373179 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-tmp-dir\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.374937 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.374911 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-dshm\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.375197 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.375179 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/830595fb-e30c-4db5-b0dc-356e37de36f8-tls-certs\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.380236 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.380216 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcc6v\" (UniqueName: \"kubernetes.io/projected/830595fb-e30c-4db5-b0dc-356e37de36f8-kube-api-access-hcc6v\") pod \"rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.498498 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.498474 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:40.777567 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:40.777505 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c"] Sep 15 11:59:40.779883 ip-10-0-128-189 kubenswrapper[2573]: W0915 11:59:40.779852 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod830595fb_e30c_4db5_b0dc_356e37de36f8.slice/crio-1b153a065b5bfe31f0bfc6a9bac4ad2a760d256263ae3a63c129ac0582ef3b94 WatchSource:0}: Error finding container 1b153a065b5bfe31f0bfc6a9bac4ad2a760d256263ae3a63c129ac0582ef3b94: Status 404 returned error can't find the container with id 1b153a065b5bfe31f0bfc6a9bac4ad2a760d256263ae3a63c129ac0582ef3b94 Sep 15 11:59:41.476931 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:41.476895 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" event={"ID":"830595fb-e30c-4db5-b0dc-356e37de36f8","Type":"ContainerStarted","Data":"55c54f3d65b00e9509e23feaf51b8de4b53ff2801c6b754c0c807b9ab19157bc"} Sep 15 11:59:41.477328 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:41.476940 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" event={"ID":"830595fb-e30c-4db5-b0dc-356e37de36f8","Type":"ContainerStarted","Data":"1b153a065b5bfe31f0bfc6a9bac4ad2a760d256263ae3a63c129ac0582ef3b94"} Sep 15 11:59:41.495253 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:41.495212 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" podStartSLOduration=1.495197703 podStartE2EDuration="1.495197703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:41.493124589 +0000 UTC m=+1368.723051124" watchObservedRunningTime="2026-09-15 11:59:41.495197703 +0000 UTC m=+1368.725124239" Sep 15 11:59:50.498822 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:50.498781 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:50.499362 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:50.498845 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:50.511200 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:50.511173 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 11:59:50.530666 ip-10-0-128-189 kubenswrapper[2573]: I0915 11:59:50.530639 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 12:00:00.162997 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.162953 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-42b26e-29824560-ggjdd"] Sep 15 12:00:00.167133 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.167109 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d2de76-29824560-d974s"] Sep 15 12:00:00.167269 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.167219 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-42b26e-29824560-ggjdd" Sep 15 12:00:00.169849 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.169828 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wjx8x\"" Sep 15 12:00:00.170489 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.170469 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d2de76-29824560-d974s" Sep 15 12:00:00.178072 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.178053 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-42b26e-29824560-ggjdd" Sep 15 12:00:00.185958 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.185887 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-42b26e-29824560-ggjdd"] Sep 15 12:00:00.187674 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.187651 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d2de76-29824560-d974s" Sep 15 12:00:00.188284 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.188258 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d2de76-29824560-d974s"] Sep 15 12:00:00.471483 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.471457 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-42b26e-29824560-ggjdd"] Sep 15 12:00:00.473248 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.473223 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d2de76-29824560-d974s"] Sep 15 12:00:00.473673 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:00:00.473647 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501c647e_fdfb_45ad_a789_ffb878d4bbef.slice/crio-dc506acd1cfee5d182261e86dd6f31707ca95d71c927d12f5201154728e0f7b4 WatchSource:0}: Error finding container dc506acd1cfee5d182261e86dd6f31707ca95d71c927d12f5201154728e0f7b4: Status 404 returned error can't find the container with id dc506acd1cfee5d182261e86dd6f31707ca95d71c927d12f5201154728e0f7b4 Sep 15 12:00:00.474309 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:00:00.474259 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb66bd79_e33f_4c69_ab81_04e5c008541b.slice/crio-defc21470919f548a4e404dabff0871185ff19015931424595b7e9aba5a4f639 WatchSource:0}: Error finding container defc21470919f548a4e404dabff0871185ff19015931424595b7e9aba5a4f639: Status 404 returned error can't find the container with id defc21470919f548a4e404dabff0871185ff19015931424595b7e9aba5a4f639 Sep 15 12:00:00.475743 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.475724 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 12:00:00.567483 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.567446 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-42b26e-29824560-ggjdd" event={"ID":"501c647e-fdfb-45ad-a789-ffb878d4bbef","Type":"ContainerStarted","Data":"dc506acd1cfee5d182261e86dd6f31707ca95d71c927d12f5201154728e0f7b4"} Sep 15 12:00:00.568685 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:00.568652 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d2de76-29824560-d974s" event={"ID":"bb66bd79-e33f-4c69-ab81-04e5c008541b","Type":"ContainerStarted","Data":"defc21470919f548a4e404dabff0871185ff19015931424595b7e9aba5a4f639"} Sep 15 12:00:04.590592 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:04.590551 2573 generic.go:356] "Generic (PLEG): container finished" podID="501c647e-fdfb-45ad-a789-ffb878d4bbef" containerID="ca7670e6cb465b118bc76de270b5e10dc9475dbca153ea6491163efed4a67bdb" exitCode=0 Sep 15 12:00:04.591078 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:04.590642 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-42b26e-29824560-ggjdd" event={"ID":"501c647e-fdfb-45ad-a789-ffb878d4bbef","Type":"ContainerDied","Data":"ca7670e6cb465b118bc76de270b5e10dc9475dbca153ea6491163efed4a67bdb"} Sep 15 12:00:04.592233 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:04.592212 2573 generic.go:356] "Generic (PLEG): container finished" podID="bb66bd79-e33f-4c69-ab81-04e5c008541b" containerID="edf91999c9bd1da7e7ab633b5911bf9614ca9ac540adabe7793e5ff517d0a538" exitCode=0 Sep 15 12:00:04.592326 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:04.592240 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d2de76-29824560-d974s" event={"ID":"bb66bd79-e33f-4c69-ab81-04e5c008541b","Type":"ContainerDied","Data":"edf91999c9bd1da7e7ab633b5911bf9614ca9ac540adabe7793e5ff517d0a538"} Sep 15 12:00:05.740431 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:05.740408 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d2de76-29824560-d974s" Sep 15 12:00:05.759188 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:05.759165 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-42b26e-29824560-ggjdd" Sep 15 12:00:06.603290 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:06.603255 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-42b26e-29824560-ggjdd" Sep 15 12:00:06.603488 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:06.603290 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-42b26e-29824560-ggjdd" event={"ID":"501c647e-fdfb-45ad-a789-ffb878d4bbef","Type":"ContainerDied","Data":"dc506acd1cfee5d182261e86dd6f31707ca95d71c927d12f5201154728e0f7b4"} Sep 15 12:00:06.603488 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:06.603338 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc506acd1cfee5d182261e86dd6f31707ca95d71c927d12f5201154728e0f7b4" Sep 15 12:00:06.604621 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:06.604601 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d2de76-29824560-d974s" Sep 15 12:00:06.604743 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:06.604628 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d2de76-29824560-d974s" event={"ID":"bb66bd79-e33f-4c69-ab81-04e5c008541b","Type":"ContainerDied","Data":"defc21470919f548a4e404dabff0871185ff19015931424595b7e9aba5a4f639"} Sep 15 12:00:06.604743 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:06.604672 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="defc21470919f548a4e404dabff0871185ff19015931424595b7e9aba5a4f639" Sep 15 12:00:10.678948 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.678917 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh"] Sep 15 12:00:10.679326 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.679289 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb66bd79-e33f-4c69-ab81-04e5c008541b" containerName="cleanup" Sep 15 12:00:10.679326 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.679301 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb66bd79-e33f-4c69-ab81-04e5c008541b" containerName="cleanup" Sep 15 12:00:10.679326 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.679317 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="501c647e-fdfb-45ad-a789-ffb878d4bbef" containerName="cleanup" Sep 15 12:00:10.679326 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.679323 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="501c647e-fdfb-45ad-a789-ffb878d4bbef" containerName="cleanup" Sep 15 12:00:10.679457 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.679392 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="501c647e-fdfb-45ad-a789-ffb878d4bbef" containerName="cleanup" Sep 15 12:00:10.679457 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.679400 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb66bd79-e33f-4c69-ab81-04e5c008541b" containerName="cleanup" Sep 15 12:00:10.682258 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.682241 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.684897 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.684878 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-7b7b3c\"/\"openshift-service-ca.crt\"" Sep 15 12:00:10.685015 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.684929 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-7b7b3c\"/\"kube-root-ca.crt\"" Sep 15 12:00:10.685855 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.685840 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-7b7b3c\"/\"rate-test-model-7b7b3c-kserve-self-signed-certs\"" Sep 15 12:00:10.685939 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.685858 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-7b7b3c\"/\"default-dockercfg-h9z4x\"" Sep 15 12:00:10.695806 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.695781 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh"] Sep 15 12:00:10.839298 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.839262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fe5716-b4c1-48e4-a39f-ec956fd671a1-tls-certs\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.839479 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.839314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-model-cache\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.839479 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.839384 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-dshm\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.839479 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.839424 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x28sq\" (UniqueName: \"kubernetes.io/projected/18fe5716-b4c1-48e4-a39f-ec956fd671a1-kube-api-access-x28sq\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.839479 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.839463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-tmp-dir\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.839626 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.839526 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-home\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.940199 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.940097 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-home\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.940199 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.940166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fe5716-b4c1-48e4-a39f-ec956fd671a1-tls-certs\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.940444 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.940224 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-model-cache\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.940444 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.940272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-dshm\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.940444 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.940295 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x28sq\" (UniqueName: \"kubernetes.io/projected/18fe5716-b4c1-48e4-a39f-ec956fd671a1-kube-api-access-x28sq\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.940444 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.940339 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-tmp-dir\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.940681 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.940521 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-home\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.940681 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.940600 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-model-cache\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.940783 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.940697 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-tmp-dir\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.942422 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.942399 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-dshm\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.942692 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.942674 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fe5716-b4c1-48e4-a39f-ec956fd671a1-tls-certs\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.949038 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.949000 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x28sq\" (UniqueName: \"kubernetes.io/projected/18fe5716-b4c1-48e4-a39f-ec956fd671a1-kube-api-access-x28sq\") pod \"rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:10.991070 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:10.991046 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:11.269967 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:11.269939 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh"] Sep 15 12:00:11.271491 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:00:11.271463 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18fe5716_b4c1_48e4_a39f_ec956fd671a1.slice/crio-e4c8dc0e39897a2aa5c71b6a9406742a18365fa527e472b37e04151ab5b446d1 WatchSource:0}: Error finding container e4c8dc0e39897a2aa5c71b6a9406742a18365fa527e472b37e04151ab5b446d1: Status 404 returned error can't find the container with id e4c8dc0e39897a2aa5c71b6a9406742a18365fa527e472b37e04151ab5b446d1 Sep 15 12:00:11.630257 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:11.630162 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" event={"ID":"18fe5716-b4c1-48e4-a39f-ec956fd671a1","Type":"ContainerStarted","Data":"37d1f92b4bff697ad79110f9ffbf1fe4757cb53a651b83f444d6634326fc213b"} Sep 15 12:00:11.630257 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:11.630200 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" event={"ID":"18fe5716-b4c1-48e4-a39f-ec956fd671a1","Type":"ContainerStarted","Data":"e4c8dc0e39897a2aa5c71b6a9406742a18365fa527e472b37e04151ab5b446d1"} Sep 15 12:00:11.651772 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:11.649036 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" podStartSLOduration=1.649002531 podStartE2EDuration="1.649002531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:11.646788659 +0000 UTC m=+1398.876715195" watchObservedRunningTime="2026-09-15 12:00:11.649002531 +0000 UTC m=+1398.878929070" Sep 15 12:00:20.991589 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:20.991551 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:20.991973 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:20.991630 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:21.004227 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:21.004197 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:21.689506 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:21.689477 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:00:35.275050 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.274849 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk"] Sep 15 12:00:35.280433 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.280399 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" Sep 15 12:00:35.283244 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.283215 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6b0cb9\"" Sep 15 12:00:35.283776 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.283748 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6b0cb9-dockercfg-58xp9\"" Sep 15 12:00:35.285009 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.284977 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk"] Sep 15 12:00:35.302846 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.302821 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d"] Sep 15 12:00:35.306170 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.306154 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" Sep 15 12:00:35.317146 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.317118 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d"] Sep 15 12:00:35.352840 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.352812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f263b25-f851-4eea-a7da-f143fb44b655-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk\" (UID: \"8f263b25-f851-4eea-a7da-f143fb44b655\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" Sep 15 12:00:35.352994 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.352854 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pksdf\" (UniqueName: \"kubernetes.io/projected/8f263b25-f851-4eea-a7da-f143fb44b655-kube-api-access-pksdf\") pod \"payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk\" (UID: \"8f263b25-f851-4eea-a7da-f143fb44b655\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" Sep 15 12:00:35.352994 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.352891 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzjrv\" (UniqueName: \"kubernetes.io/projected/3a53555a-ac4b-4d51-9422-4f07a3894757-kube-api-access-kzjrv\") pod \"payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d\" (UID: \"3a53555a-ac4b-4d51-9422-4f07a3894757\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" Sep 15 12:00:35.352994 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.352912 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a53555a-ac4b-4d51-9422-4f07a3894757-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d\" (UID: \"3a53555a-ac4b-4d51-9422-4f07a3894757\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" Sep 15 12:00:35.454258 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.454223 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzjrv\" (UniqueName: \"kubernetes.io/projected/3a53555a-ac4b-4d51-9422-4f07a3894757-kube-api-access-kzjrv\") pod \"payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d\" (UID: \"3a53555a-ac4b-4d51-9422-4f07a3894757\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" Sep 15 12:00:35.454258 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.454258 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a53555a-ac4b-4d51-9422-4f07a3894757-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d\" (UID: \"3a53555a-ac4b-4d51-9422-4f07a3894757\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" Sep 15 12:00:35.454515 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.454337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f263b25-f851-4eea-a7da-f143fb44b655-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk\" (UID: \"8f263b25-f851-4eea-a7da-f143fb44b655\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" Sep 15 12:00:35.454515 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.454372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pksdf\" (UniqueName: \"kubernetes.io/projected/8f263b25-f851-4eea-a7da-f143fb44b655-kube-api-access-pksdf\") pod \"payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk\" (UID: \"8f263b25-f851-4eea-a7da-f143fb44b655\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" Sep 15 12:00:35.455101 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.455072 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f263b25-f851-4eea-a7da-f143fb44b655-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk\" (UID: \"8f263b25-f851-4eea-a7da-f143fb44b655\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" Sep 15 12:00:35.455189 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.455072 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a53555a-ac4b-4d51-9422-4f07a3894757-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d\" (UID: \"3a53555a-ac4b-4d51-9422-4f07a3894757\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" Sep 15 12:00:35.465777 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.465750 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzjrv\" (UniqueName: \"kubernetes.io/projected/3a53555a-ac4b-4d51-9422-4f07a3894757-kube-api-access-kzjrv\") pod \"payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d\" (UID: \"3a53555a-ac4b-4d51-9422-4f07a3894757\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" Sep 15 12:00:35.465912 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.465885 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pksdf\" (UniqueName: \"kubernetes.io/projected/8f263b25-f851-4eea-a7da-f143fb44b655-kube-api-access-pksdf\") pod \"payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk\" (UID: \"8f263b25-f851-4eea-a7da-f143fb44b655\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" Sep 15 12:00:35.592407 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.592315 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" Sep 15 12:00:35.617356 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.617321 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" Sep 15 12:00:35.866446 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.866411 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk"] Sep 15 12:00:35.868417 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:00:35.868393 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f263b25_f851_4eea_a7da_f143fb44b655.slice/crio-dc9bf12fd5e2e6ee098f7d787e4cff496ce8fdddae3db92078ca7d81f5d92ebf WatchSource:0}: Error finding container dc9bf12fd5e2e6ee098f7d787e4cff496ce8fdddae3db92078ca7d81f5d92ebf: Status 404 returned error can't find the container with id dc9bf12fd5e2e6ee098f7d787e4cff496ce8fdddae3db92078ca7d81f5d92ebf Sep 15 12:00:35.903690 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:35.903659 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d"] Sep 15 12:00:35.905536 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:00:35.905506 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a53555a_ac4b_4d51_9422_4f07a3894757.slice/crio-55e801ae3d50622060415a14458f31bdb06cc5cd6409a0aef56ad6081a539ab5 WatchSource:0}: Error finding container 55e801ae3d50622060415a14458f31bdb06cc5cd6409a0aef56ad6081a539ab5: Status 404 returned error can't find the container with id 55e801ae3d50622060415a14458f31bdb06cc5cd6409a0aef56ad6081a539ab5 Sep 15 12:00:36.756980 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:36.756936 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" event={"ID":"3a53555a-ac4b-4d51-9422-4f07a3894757","Type":"ContainerStarted","Data":"46edf88670659e7fdd2c0f3f9f63360a482219d1b10fcc3e7aafa5af4e7375c7"} Sep 15 12:00:36.757439 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:36.757008 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" event={"ID":"3a53555a-ac4b-4d51-9422-4f07a3894757","Type":"ContainerStarted","Data":"55e801ae3d50622060415a14458f31bdb06cc5cd6409a0aef56ad6081a539ab5"} Sep 15 12:00:36.757439 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:36.757060 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" Sep 15 12:00:36.758660 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:36.758633 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" event={"ID":"8f263b25-f851-4eea-a7da-f143fb44b655","Type":"ContainerStarted","Data":"25635eefc9a101ae9fa17bebd688a44d7336f61312173cc39ec47201a4514593"} Sep 15 12:00:36.758777 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:36.758680 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" event={"ID":"8f263b25-f851-4eea-a7da-f143fb44b655","Type":"ContainerStarted","Data":"dc9bf12fd5e2e6ee098f7d787e4cff496ce8fdddae3db92078ca7d81f5d92ebf"} Sep 15 12:00:36.758777 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:36.758705 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" Sep 15 12:00:36.793241 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:36.793191 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" podStartSLOduration=1.79317734 podStartE2EDuration="1.79317734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:36.775084024 +0000 UTC m=+1424.005010560" watchObservedRunningTime="2026-09-15 12:00:36.79317734 +0000 UTC m=+1424.023103877" Sep 15 12:00:47.765760 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:47.765678 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" Sep 15 12:00:47.766140 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:47.765872 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" Sep 15 12:00:47.787894 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:00:47.787843 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" podStartSLOduration=12.787826652 podStartE2EDuration="12.787826652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:36.791958405 +0000 UTC m=+1424.021884942" watchObservedRunningTime="2026-09-15 12:00:47.787826652 +0000 UTC m=+1435.017753187" Sep 15 12:01:21.027801 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:21.027766 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c"] Sep 15 12:01:21.851364 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:21.851327 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh"] Sep 15 12:01:36.046344 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.046297 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" podUID="830595fb-e30c-4db5-b0dc-356e37de36f8" containerName="main" containerID="cri-o://55c54f3d65b00e9509e23feaf51b8de4b53ff2801c6b754c0c807b9ab19157bc" gracePeriod=45 Sep 15 12:01:36.291772 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.291747 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 12:01:36.391073 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.390965 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcc6v\" (UniqueName: \"kubernetes.io/projected/830595fb-e30c-4db5-b0dc-356e37de36f8-kube-api-access-hcc6v\") pod \"830595fb-e30c-4db5-b0dc-356e37de36f8\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " Sep 15 12:01:36.391073 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.391019 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-tmp-dir\") pod \"830595fb-e30c-4db5-b0dc-356e37de36f8\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " Sep 15 12:01:36.391284 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.391136 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-model-cache\") pod \"830595fb-e30c-4db5-b0dc-356e37de36f8\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " Sep 15 12:01:36.391284 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.391187 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-dshm\") pod \"830595fb-e30c-4db5-b0dc-356e37de36f8\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " Sep 15 12:01:36.391284 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.391209 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-home\") pod \"830595fb-e30c-4db5-b0dc-356e37de36f8\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " Sep 15 12:01:36.391284 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.391232 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/830595fb-e30c-4db5-b0dc-356e37de36f8-tls-certs\") pod \"830595fb-e30c-4db5-b0dc-356e37de36f8\" (UID: \"830595fb-e30c-4db5-b0dc-356e37de36f8\") " Sep 15 12:01:36.391478 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.391357 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "830595fb-e30c-4db5-b0dc-356e37de36f8" (UID: "830595fb-e30c-4db5-b0dc-356e37de36f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:01:36.391478 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.391424 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "830595fb-e30c-4db5-b0dc-356e37de36f8" (UID: "830595fb-e30c-4db5-b0dc-356e37de36f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:01:36.391478 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.391461 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-home" (OuterVolumeSpecName: "home") pod "830595fb-e30c-4db5-b0dc-356e37de36f8" (UID: "830595fb-e30c-4db5-b0dc-356e37de36f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:01:36.391607 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.391587 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-home\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:36.391607 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.391602 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-tmp-dir\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:36.391719 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.391612 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-model-cache\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:36.393191 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.393166 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830595fb-e30c-4db5-b0dc-356e37de36f8-kube-api-access-hcc6v" (OuterVolumeSpecName: "kube-api-access-hcc6v") pod "830595fb-e30c-4db5-b0dc-356e37de36f8" (UID: "830595fb-e30c-4db5-b0dc-356e37de36f8"). InnerVolumeSpecName "kube-api-access-hcc6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:01:36.393288 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.393219 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-dshm" (OuterVolumeSpecName: "dshm") pod "830595fb-e30c-4db5-b0dc-356e37de36f8" (UID: "830595fb-e30c-4db5-b0dc-356e37de36f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:01:36.393351 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.393328 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830595fb-e30c-4db5-b0dc-356e37de36f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "830595fb-e30c-4db5-b0dc-356e37de36f8" (UID: "830595fb-e30c-4db5-b0dc-356e37de36f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:01:36.492786 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.492755 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/830595fb-e30c-4db5-b0dc-356e37de36f8-dshm\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:36.492786 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.492781 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/830595fb-e30c-4db5-b0dc-356e37de36f8-tls-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:36.492944 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.492797 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcc6v\" (UniqueName: \"kubernetes.io/projected/830595fb-e30c-4db5-b0dc-356e37de36f8-kube-api-access-hcc6v\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:36.871076 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:36.870994 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" podUID="18fe5716-b4c1-48e4-a39f-ec956fd671a1" containerName="main" containerID="cri-o://37d1f92b4bff697ad79110f9ffbf1fe4757cb53a651b83f444d6634326fc213b" gracePeriod=45 Sep 15 12:01:37.068989 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.068962 2573 generic.go:356] "Generic (PLEG): container finished" podID="18fe5716-b4c1-48e4-a39f-ec956fd671a1" containerID="37d1f92b4bff697ad79110f9ffbf1fe4757cb53a651b83f444d6634326fc213b" exitCode=0 Sep 15 12:01:37.069406 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.069065 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" event={"ID":"18fe5716-b4c1-48e4-a39f-ec956fd671a1","Type":"ContainerDied","Data":"37d1f92b4bff697ad79110f9ffbf1fe4757cb53a651b83f444d6634326fc213b"} Sep 15 12:01:37.070609 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.070584 2573 generic.go:356] "Generic (PLEG): container finished" podID="830595fb-e30c-4db5-b0dc-356e37de36f8" containerID="55c54f3d65b00e9509e23feaf51b8de4b53ff2801c6b754c0c807b9ab19157bc" exitCode=0 Sep 15 12:01:37.070722 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.070639 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" Sep 15 12:01:37.070722 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.070673 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" event={"ID":"830595fb-e30c-4db5-b0dc-356e37de36f8","Type":"ContainerDied","Data":"55c54f3d65b00e9509e23feaf51b8de4b53ff2801c6b754c0c807b9ab19157bc"} Sep 15 12:01:37.070817 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.070723 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c" event={"ID":"830595fb-e30c-4db5-b0dc-356e37de36f8","Type":"ContainerDied","Data":"1b153a065b5bfe31f0bfc6a9bac4ad2a760d256263ae3a63c129ac0582ef3b94"} Sep 15 12:01:37.070817 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.070754 2573 scope.go:117] "RemoveContainer" containerID="55c54f3d65b00e9509e23feaf51b8de4b53ff2801c6b754c0c807b9ab19157bc" Sep 15 12:01:37.080781 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.080761 2573 scope.go:117] "RemoveContainer" containerID="55c54f3d65b00e9509e23feaf51b8de4b53ff2801c6b754c0c807b9ab19157bc" Sep 15 12:01:37.081120 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:01:37.081092 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c54f3d65b00e9509e23feaf51b8de4b53ff2801c6b754c0c807b9ab19157bc\": container with ID starting with 55c54f3d65b00e9509e23feaf51b8de4b53ff2801c6b754c0c807b9ab19157bc not found: ID does not exist" containerID="55c54f3d65b00e9509e23feaf51b8de4b53ff2801c6b754c0c807b9ab19157bc" Sep 15 12:01:37.081198 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.081133 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c54f3d65b00e9509e23feaf51b8de4b53ff2801c6b754c0c807b9ab19157bc"} err="failed to get container status \"55c54f3d65b00e9509e23feaf51b8de4b53ff2801c6b754c0c807b9ab19157bc\": rpc error: code = NotFound desc = could not find container \"55c54f3d65b00e9509e23feaf51b8de4b53ff2801c6b754c0c807b9ab19157bc\": container with ID starting with 55c54f3d65b00e9509e23feaf51b8de4b53ff2801c6b754c0c807b9ab19157bc not found: ID does not exist" Sep 15 12:01:37.105461 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.105434 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c"] Sep 15 12:01:37.108241 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.108218 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-42b26e/rate-test-model-42b26e-kserve-64dc7f7bbc-wmg6c"] Sep 15 12:01:37.126895 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.126831 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:01:37.199382 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.199353 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fe5716-b4c1-48e4-a39f-ec956fd671a1-tls-certs\") pod \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " Sep 15 12:01:37.199569 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.199436 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x28sq\" (UniqueName: \"kubernetes.io/projected/18fe5716-b4c1-48e4-a39f-ec956fd671a1-kube-api-access-x28sq\") pod \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " Sep 15 12:01:37.199569 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.199471 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-dshm\") pod \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " Sep 15 12:01:37.199569 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.199498 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-model-cache\") pod \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " Sep 15 12:01:37.199745 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.199575 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-tmp-dir\") pod \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " Sep 15 12:01:37.199745 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.199629 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-home\") pod \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\" (UID: \"18fe5716-b4c1-48e4-a39f-ec956fd671a1\") " Sep 15 12:01:37.199842 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.199788 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "18fe5716-b4c1-48e4-a39f-ec956fd671a1" (UID: "18fe5716-b4c1-48e4-a39f-ec956fd671a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:01:37.199993 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.199975 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-model-cache\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:37.200073 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.200005 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-home" (OuterVolumeSpecName: "home") pod "18fe5716-b4c1-48e4-a39f-ec956fd671a1" (UID: "18fe5716-b4c1-48e4-a39f-ec956fd671a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:01:37.200170 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.200150 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18fe5716-b4c1-48e4-a39f-ec956fd671a1" (UID: "18fe5716-b4c1-48e4-a39f-ec956fd671a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:01:37.201563 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.201538 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fe5716-b4c1-48e4-a39f-ec956fd671a1-kube-api-access-x28sq" (OuterVolumeSpecName: "kube-api-access-x28sq") pod "18fe5716-b4c1-48e4-a39f-ec956fd671a1" (UID: "18fe5716-b4c1-48e4-a39f-ec956fd671a1"). InnerVolumeSpecName "kube-api-access-x28sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:01:37.201955 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.201934 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-dshm" (OuterVolumeSpecName: "dshm") pod "18fe5716-b4c1-48e4-a39f-ec956fd671a1" (UID: "18fe5716-b4c1-48e4-a39f-ec956fd671a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:01:37.201955 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.201945 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18fe5716-b4c1-48e4-a39f-ec956fd671a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18fe5716-b4c1-48e4-a39f-ec956fd671a1" (UID: "18fe5716-b4c1-48e4-a39f-ec956fd671a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:01:37.235271 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.235243 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="830595fb-e30c-4db5-b0dc-356e37de36f8" path="/var/lib/kubelet/pods/830595fb-e30c-4db5-b0dc-356e37de36f8/volumes" Sep 15 12:01:37.301313 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.301285 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x28sq\" (UniqueName: \"kubernetes.io/projected/18fe5716-b4c1-48e4-a39f-ec956fd671a1-kube-api-access-x28sq\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:37.301313 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.301311 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-dshm\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:37.301449 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.301321 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-tmp-dir\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:37.301449 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.301332 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fe5716-b4c1-48e4-a39f-ec956fd671a1-home\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:37.301449 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:37.301339 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fe5716-b4c1-48e4-a39f-ec956fd671a1-tls-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:38.077519 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:38.077482 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" event={"ID":"18fe5716-b4c1-48e4-a39f-ec956fd671a1","Type":"ContainerDied","Data":"e4c8dc0e39897a2aa5c71b6a9406742a18365fa527e472b37e04151ab5b446d1"} Sep 15 12:01:38.077519 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:38.077512 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh" Sep 15 12:01:38.077994 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:38.077535 2573 scope.go:117] "RemoveContainer" containerID="37d1f92b4bff697ad79110f9ffbf1fe4757cb53a651b83f444d6634326fc213b" Sep 15 12:01:38.098955 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:38.098929 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh"] Sep 15 12:01:38.101863 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:38.101834 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7b7b3c/rate-test-model-7b7b3c-kserve-58c798dc5-kl5sh"] Sep 15 12:01:39.240177 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:39.240144 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18fe5716-b4c1-48e4-a39f-ec956fd671a1" path="/var/lib/kubelet/pods/18fe5716-b4c1-48e4-a39f-ec956fd671a1/volumes" Sep 15 12:01:45.041227 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.041192 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d"] Sep 15 12:01:45.041828 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.041456 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" podUID="3a53555a-ac4b-4d51-9422-4f07a3894757" containerName="payload-processing" containerID="cri-o://46edf88670659e7fdd2c0f3f9f63360a482219d1b10fcc3e7aafa5af4e7375c7" gracePeriod=30 Sep 15 12:01:45.049301 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.049199 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk"] Sep 15 12:01:45.049460 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.049429 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" podUID="8f263b25-f851-4eea-a7da-f143fb44b655" containerName="payload-pre-processing" containerID="cri-o://25635eefc9a101ae9fa17bebd688a44d7336f61312173cc39ec47201a4514593" gracePeriod=30 Sep 15 12:01:45.296163 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.296110 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" Sep 15 12:01:45.326004 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.325976 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" Sep 15 12:01:45.380904 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.380868 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a53555a-ac4b-4d51-9422-4f07a3894757-plugins-config-volume\") pod \"3a53555a-ac4b-4d51-9422-4f07a3894757\" (UID: \"3a53555a-ac4b-4d51-9422-4f07a3894757\") " Sep 15 12:01:45.381119 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.380922 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzjrv\" (UniqueName: \"kubernetes.io/projected/3a53555a-ac4b-4d51-9422-4f07a3894757-kube-api-access-kzjrv\") pod \"3a53555a-ac4b-4d51-9422-4f07a3894757\" (UID: \"3a53555a-ac4b-4d51-9422-4f07a3894757\") " Sep 15 12:01:45.381459 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.381435 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a53555a-ac4b-4d51-9422-4f07a3894757-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a53555a-ac4b-4d51-9422-4f07a3894757" (UID: "3a53555a-ac4b-4d51-9422-4f07a3894757"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:01:45.383240 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.383214 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a53555a-ac4b-4d51-9422-4f07a3894757-kube-api-access-kzjrv" (OuterVolumeSpecName: "kube-api-access-kzjrv") pod "3a53555a-ac4b-4d51-9422-4f07a3894757" (UID: "3a53555a-ac4b-4d51-9422-4f07a3894757"). InnerVolumeSpecName "kube-api-access-kzjrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:01:45.481910 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.481878 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f263b25-f851-4eea-a7da-f143fb44b655-plugins-config-volume\") pod \"8f263b25-f851-4eea-a7da-f143fb44b655\" (UID: \"8f263b25-f851-4eea-a7da-f143fb44b655\") " Sep 15 12:01:45.482096 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.481948 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pksdf\" (UniqueName: \"kubernetes.io/projected/8f263b25-f851-4eea-a7da-f143fb44b655-kube-api-access-pksdf\") pod \"8f263b25-f851-4eea-a7da-f143fb44b655\" (UID: \"8f263b25-f851-4eea-a7da-f143fb44b655\") " Sep 15 12:01:45.482213 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.482193 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a53555a-ac4b-4d51-9422-4f07a3894757-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:45.482255 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.482219 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzjrv\" (UniqueName: \"kubernetes.io/projected/3a53555a-ac4b-4d51-9422-4f07a3894757-kube-api-access-kzjrv\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:45.482312 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.482279 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f263b25-f851-4eea-a7da-f143fb44b655-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f263b25-f851-4eea-a7da-f143fb44b655" (UID: "8f263b25-f851-4eea-a7da-f143fb44b655"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:01:45.483955 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.483937 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f263b25-f851-4eea-a7da-f143fb44b655-kube-api-access-pksdf" (OuterVolumeSpecName: "kube-api-access-pksdf") pod "8f263b25-f851-4eea-a7da-f143fb44b655" (UID: "8f263b25-f851-4eea-a7da-f143fb44b655"). InnerVolumeSpecName "kube-api-access-pksdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:01:45.583134 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.583062 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f263b25-f851-4eea-a7da-f143fb44b655-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:45.583134 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:45.583087 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pksdf\" (UniqueName: \"kubernetes.io/projected/8f263b25-f851-4eea-a7da-f143fb44b655-kube-api-access-pksdf\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:46.119456 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.119418 2573 generic.go:356] "Generic (PLEG): container finished" podID="3a53555a-ac4b-4d51-9422-4f07a3894757" containerID="46edf88670659e7fdd2c0f3f9f63360a482219d1b10fcc3e7aafa5af4e7375c7" exitCode=0 Sep 15 12:01:46.119883 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.119496 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" Sep 15 12:01:46.119883 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.119509 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" event={"ID":"3a53555a-ac4b-4d51-9422-4f07a3894757","Type":"ContainerDied","Data":"46edf88670659e7fdd2c0f3f9f63360a482219d1b10fcc3e7aafa5af4e7375c7"} Sep 15 12:01:46.119883 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.119593 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d" event={"ID":"3a53555a-ac4b-4d51-9422-4f07a3894757","Type":"ContainerDied","Data":"55e801ae3d50622060415a14458f31bdb06cc5cd6409a0aef56ad6081a539ab5"} Sep 15 12:01:46.119883 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.119613 2573 scope.go:117] "RemoveContainer" containerID="46edf88670659e7fdd2c0f3f9f63360a482219d1b10fcc3e7aafa5af4e7375c7" Sep 15 12:01:46.121109 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.121085 2573 generic.go:356] "Generic (PLEG): container finished" podID="8f263b25-f851-4eea-a7da-f143fb44b655" containerID="25635eefc9a101ae9fa17bebd688a44d7336f61312173cc39ec47201a4514593" exitCode=0 Sep 15 12:01:46.121215 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.121134 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" event={"ID":"8f263b25-f851-4eea-a7da-f143fb44b655","Type":"ContainerDied","Data":"25635eefc9a101ae9fa17bebd688a44d7336f61312173cc39ec47201a4514593"} Sep 15 12:01:46.121215 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.121145 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" Sep 15 12:01:46.121215 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.121196 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk" event={"ID":"8f263b25-f851-4eea-a7da-f143fb44b655","Type":"ContainerDied","Data":"dc9bf12fd5e2e6ee098f7d787e4cff496ce8fdddae3db92078ca7d81f5d92ebf"} Sep 15 12:01:46.133322 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.133304 2573 scope.go:117] "RemoveContainer" containerID="46edf88670659e7fdd2c0f3f9f63360a482219d1b10fcc3e7aafa5af4e7375c7" Sep 15 12:01:46.133621 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:01:46.133601 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46edf88670659e7fdd2c0f3f9f63360a482219d1b10fcc3e7aafa5af4e7375c7\": container with ID starting with 46edf88670659e7fdd2c0f3f9f63360a482219d1b10fcc3e7aafa5af4e7375c7 not found: ID does not exist" containerID="46edf88670659e7fdd2c0f3f9f63360a482219d1b10fcc3e7aafa5af4e7375c7" Sep 15 12:01:46.133672 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.133629 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46edf88670659e7fdd2c0f3f9f63360a482219d1b10fcc3e7aafa5af4e7375c7"} err="failed to get container status \"46edf88670659e7fdd2c0f3f9f63360a482219d1b10fcc3e7aafa5af4e7375c7\": rpc error: code = NotFound desc = could not find container \"46edf88670659e7fdd2c0f3f9f63360a482219d1b10fcc3e7aafa5af4e7375c7\": container with ID starting with 46edf88670659e7fdd2c0f3f9f63360a482219d1b10fcc3e7aafa5af4e7375c7 not found: ID does not exist" Sep 15 12:01:46.133672 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.133648 2573 scope.go:117] "RemoveContainer" containerID="25635eefc9a101ae9fa17bebd688a44d7336f61312173cc39ec47201a4514593" Sep 15 12:01:46.141236 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.141220 2573 scope.go:117] "RemoveContainer" containerID="25635eefc9a101ae9fa17bebd688a44d7336f61312173cc39ec47201a4514593" Sep 15 12:01:46.141490 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:01:46.141473 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25635eefc9a101ae9fa17bebd688a44d7336f61312173cc39ec47201a4514593\": container with ID starting with 25635eefc9a101ae9fa17bebd688a44d7336f61312173cc39ec47201a4514593 not found: ID does not exist" containerID="25635eefc9a101ae9fa17bebd688a44d7336f61312173cc39ec47201a4514593" Sep 15 12:01:46.141534 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.141496 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25635eefc9a101ae9fa17bebd688a44d7336f61312173cc39ec47201a4514593"} err="failed to get container status \"25635eefc9a101ae9fa17bebd688a44d7336f61312173cc39ec47201a4514593\": rpc error: code = NotFound desc = could not find container \"25635eefc9a101ae9fa17bebd688a44d7336f61312173cc39ec47201a4514593\": container with ID starting with 25635eefc9a101ae9fa17bebd688a44d7336f61312173cc39ec47201a4514593 not found: ID does not exist" Sep 15 12:01:46.148826 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.148804 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk"] Sep 15 12:01:46.153089 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.153068 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b0cb9-5758b8cf7d-5zmtk"] Sep 15 12:01:46.165488 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.165469 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d"] Sep 15 12:01:46.170553 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:46.170534 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b0cb9-88bc7c597-xwc9d"] Sep 15 12:01:47.234598 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:47.234564 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a53555a-ac4b-4d51-9422-4f07a3894757" path="/var/lib/kubelet/pods/3a53555a-ac4b-4d51-9422-4f07a3894757/volumes" Sep 15 12:01:47.234979 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:47.234933 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f263b25-f851-4eea-a7da-f143fb44b655" path="/var/lib/kubelet/pods/8f263b25-f851-4eea-a7da-f143fb44b655/volumes" Sep 15 12:01:53.300709 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.300682 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 12:01:53.305744 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.305722 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 12:01:53.308597 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.308565 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 12:01:53.313508 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.313492 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 12:01:53.978034 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.977985 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6"] Sep 15 12:01:53.978439 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.978423 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="830595fb-e30c-4db5-b0dc-356e37de36f8" containerName="main" Sep 15 12:01:53.978439 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.978441 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="830595fb-e30c-4db5-b0dc-356e37de36f8" containerName="main" Sep 15 12:01:53.978517 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.978449 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f263b25-f851-4eea-a7da-f143fb44b655" containerName="payload-pre-processing" Sep 15 12:01:53.978517 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.978455 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f263b25-f851-4eea-a7da-f143fb44b655" containerName="payload-pre-processing" Sep 15 12:01:53.978517 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.978465 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a53555a-ac4b-4d51-9422-4f07a3894757" containerName="payload-processing" Sep 15 12:01:53.978517 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.978471 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a53555a-ac4b-4d51-9422-4f07a3894757" containerName="payload-processing" Sep 15 12:01:53.978517 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.978480 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fe5716-b4c1-48e4-a39f-ec956fd671a1" containerName="main" Sep 15 12:01:53.978517 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.978486 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fe5716-b4c1-48e4-a39f-ec956fd671a1" containerName="main" Sep 15 12:01:53.978700 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.978557 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="18fe5716-b4c1-48e4-a39f-ec956fd671a1" containerName="main" Sep 15 12:01:53.978700 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.978565 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="830595fb-e30c-4db5-b0dc-356e37de36f8" containerName="main" Sep 15 12:01:53.978700 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.978572 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a53555a-ac4b-4d51-9422-4f07a3894757" containerName="payload-processing" Sep 15 12:01:53.978700 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.978580 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f263b25-f851-4eea-a7da-f143fb44b655" containerName="payload-pre-processing" Sep 15 12:01:53.983440 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.983421 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6" Sep 15 12:01:53.986867 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.986841 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wjx8x\"" Sep 15 12:01:53.987187 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.987171 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:01:53.994119 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:53.994091 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6"] Sep 15 12:01:54.160640 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:54.160605 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6\" (UID: \"fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6" Sep 15 12:01:54.261058 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:54.261012 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6\" (UID: \"fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6" Sep 15 12:01:54.261602 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:54.261582 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6\" (UID: \"fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6" Sep 15 12:01:54.296011 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:54.295983 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6" Sep 15 12:01:54.579253 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:54.579223 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6"] Sep 15 12:01:54.581460 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:01:54.581429 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5f0345_ae67_4bbf_ace2_f8a0e3e8b9d3.slice/crio-c313bae5fb25753c3ba5ab149b3025987da69a2edf2dea75f9cb13d9f86b35ed WatchSource:0}: Error finding container c313bae5fb25753c3ba5ab149b3025987da69a2edf2dea75f9cb13d9f86b35ed: Status 404 returned error can't find the container with id c313bae5fb25753c3ba5ab149b3025987da69a2edf2dea75f9cb13d9f86b35ed Sep 15 12:01:55.170357 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:55.170262 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6" event={"ID":"fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3","Type":"ContainerStarted","Data":"538b1caed64a876881b7a60a49660c0e4d93ccc00b8c3ea35c5118f6aa8afa98"} Sep 15 12:01:55.170357 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:55.170333 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6" event={"ID":"fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3","Type":"ContainerStarted","Data":"c313bae5fb25753c3ba5ab149b3025987da69a2edf2dea75f9cb13d9f86b35ed"} Sep 15 12:01:55.202221 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:55.202173 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6" podStartSLOduration=2.202157426 podStartE2EDuration="2.202157426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:01:55.200076627 +0000 UTC m=+1502.430003163" watchObservedRunningTime="2026-09-15 12:01:55.202157426 +0000 UTC m=+1502.432084009" Sep 15 12:01:56.177248 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:56.177210 2573 generic.go:356] "Generic (PLEG): container finished" podID="fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3" containerID="538b1caed64a876881b7a60a49660c0e4d93ccc00b8c3ea35c5118f6aa8afa98" exitCode=0 Sep 15 12:01:56.177758 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:56.177256 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6" event={"ID":"fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3","Type":"ContainerDied","Data":"538b1caed64a876881b7a60a49660c0e4d93ccc00b8c3ea35c5118f6aa8afa98"} Sep 15 12:01:57.313088 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:57.313063 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6" Sep 15 12:01:57.488596 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:57.488512 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3-maas-api-service-ca\") pod \"fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3\" (UID: \"fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3\") " Sep 15 12:01:57.488848 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:57.488826 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3" (UID: "fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:01:57.590106 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:57.590074 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3-maas-api-service-ca\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:01:58.187963 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:58.187933 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6" Sep 15 12:01:58.188147 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:58.187977 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6" event={"ID":"fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3","Type":"ContainerDied","Data":"c313bae5fb25753c3ba5ab149b3025987da69a2edf2dea75f9cb13d9f86b35ed"} Sep 15 12:01:58.188147 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:58.188069 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c313bae5fb25753c3ba5ab149b3025987da69a2edf2dea75f9cb13d9f86b35ed" Sep 15 12:01:59.881925 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:59.881892 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc"] Sep 15 12:01:59.882412 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:59.882269 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3" containerName="revoke-keys" Sep 15 12:01:59.882412 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:59.882280 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3" containerName="revoke-keys" Sep 15 12:01:59.882412 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:59.882349 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3" containerName="revoke-keys" Sep 15 12:01:59.885562 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:59.885538 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:01:59.889137 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:59.889111 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 15 12:01:59.889918 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:59.889896 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:01:59.890014 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:59.889925 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 15 12:01:59.890110 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:59.890013 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0249d8-serving-cert\"" Sep 15 12:01:59.890169 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:59.890139 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9dhbv\"" Sep 15 12:01:59.895918 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:01:59.895896 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc"] Sep 15 12:02:00.011136 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.011097 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfz5d\" (UniqueName: \"kubernetes.io/projected/7ba9e261-7244-404f-9a2a-fcb7d615858a-kube-api-access-cfz5d\") pod \"maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:00.011321 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.011170 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ba9e261-7244-404f-9a2a-fcb7d615858a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:00.011321 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.011198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba9e261-7244-404f-9a2a-fcb7d615858a-metrics-certs\") pod \"maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:00.011321 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.011297 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba9e261-7244-404f-9a2a-fcb7d615858a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:00.111769 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.111728 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba9e261-7244-404f-9a2a-fcb7d615858a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:00.111976 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.111790 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfz5d\" (UniqueName: \"kubernetes.io/projected/7ba9e261-7244-404f-9a2a-fcb7d615858a-kube-api-access-cfz5d\") pod \"maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:00.111976 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.111839 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ba9e261-7244-404f-9a2a-fcb7d615858a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:00.111976 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.111863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba9e261-7244-404f-9a2a-fcb7d615858a-metrics-certs\") pod \"maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:00.112432 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.112409 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba9e261-7244-404f-9a2a-fcb7d615858a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:00.114302 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.114270 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba9e261-7244-404f-9a2a-fcb7d615858a-metrics-certs\") pod \"maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:00.114401 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.114332 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ba9e261-7244-404f-9a2a-fcb7d615858a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:00.120542 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.120520 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfz5d\" (UniqueName: \"kubernetes.io/projected/7ba9e261-7244-404f-9a2a-fcb7d615858a-kube-api-access-cfz5d\") pod \"maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:00.198071 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.197963 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:00.474273 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.474246 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc"] Sep 15 12:02:00.476664 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:02:00.476624 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba9e261_7244_404f_9a2a_fcb7d615858a.slice/crio-b85aa6949169b6189beb044ab755342ef8d0a83c5c346d9a05139099c0a4ff3d WatchSource:0}: Error finding container b85aa6949169b6189beb044ab755342ef8d0a83c5c346d9a05139099c0a4ff3d: Status 404 returned error can't find the container with id b85aa6949169b6189beb044ab755342ef8d0a83c5c346d9a05139099c0a4ff3d Sep 15 12:02:00.684624 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.684584 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57"] Sep 15 12:02:00.689230 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.689213 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" Sep 15 12:02:00.691678 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.691656 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0249d8\"" Sep 15 12:02:00.691805 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.691733 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0249d8-dockercfg-8w799\"" Sep 15 12:02:00.697658 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.697632 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57"] Sep 15 12:02:00.818969 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.818936 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6298a16-934c-4717-81e1-78763106cf27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57\" (UID: \"d6298a16-934c-4717-81e1-78763106cf27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" Sep 15 12:02:00.819171 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.819052 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khmdh\" (UniqueName: \"kubernetes.io/projected/d6298a16-934c-4717-81e1-78763106cf27-kube-api-access-khmdh\") pod \"payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57\" (UID: \"d6298a16-934c-4717-81e1-78763106cf27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" Sep 15 12:02:00.919856 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.919824 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khmdh\" (UniqueName: \"kubernetes.io/projected/d6298a16-934c-4717-81e1-78763106cf27-kube-api-access-khmdh\") pod \"payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57\" (UID: \"d6298a16-934c-4717-81e1-78763106cf27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" Sep 15 12:02:00.920239 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.919879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6298a16-934c-4717-81e1-78763106cf27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57\" (UID: \"d6298a16-934c-4717-81e1-78763106cf27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" Sep 15 12:02:00.920556 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.920533 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6298a16-934c-4717-81e1-78763106cf27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57\" (UID: \"d6298a16-934c-4717-81e1-78763106cf27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" Sep 15 12:02:00.927456 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:00.927437 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khmdh\" (UniqueName: \"kubernetes.io/projected/d6298a16-934c-4717-81e1-78763106cf27-kube-api-access-khmdh\") pod \"payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57\" (UID: \"d6298a16-934c-4717-81e1-78763106cf27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" Sep 15 12:02:01.002270 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:01.002226 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" Sep 15 12:02:01.204466 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:01.204370 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" event={"ID":"7ba9e261-7244-404f-9a2a-fcb7d615858a","Type":"ContainerStarted","Data":"a5356f49017ba1b0c5b10df450fc7afe5a7224b89794f13591eb8ad12dfc6674"} Sep 15 12:02:01.204466 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:01.204436 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" event={"ID":"7ba9e261-7244-404f-9a2a-fcb7d615858a","Type":"ContainerStarted","Data":"b85aa6949169b6189beb044ab755342ef8d0a83c5c346d9a05139099c0a4ff3d"} Sep 15 12:02:01.204692 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:01.204553 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:01.225814 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:01.225766 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" podStartSLOduration=1.8368169829999998 podStartE2EDuration="2.225749947s" totalImagesPullingTime="388.932964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:01.223796456 +0000 UTC m=+1508.453723003" watchObservedRunningTime="2026-09-15 12:02:01.225749947 +0000 UTC m=+1508.455676494" Sep 15 12:02:01.272055 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:01.272018 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57"] Sep 15 12:02:01.273943 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:02:01.273908 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6298a16_934c_4717_81e1_78763106cf27.slice/crio-1076f47cf3cf60257f094422d0513fb8a51a3ef65ee0a20b2dfe03f8a0fc1017 WatchSource:0}: Error finding container 1076f47cf3cf60257f094422d0513fb8a51a3ef65ee0a20b2dfe03f8a0fc1017: Status 404 returned error can't find the container with id 1076f47cf3cf60257f094422d0513fb8a51a3ef65ee0a20b2dfe03f8a0fc1017 Sep 15 12:02:02.211334 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:02.211291 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" event={"ID":"d6298a16-934c-4717-81e1-78763106cf27","Type":"ContainerStarted","Data":"7345b6189a5474ed0ee4a3ec451be3dc0f4ac6ff60546e941252a15f32dbffee"} Sep 15 12:02:02.211722 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:02.211388 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" event={"ID":"d6298a16-934c-4717-81e1-78763106cf27","Type":"ContainerStarted","Data":"1076f47cf3cf60257f094422d0513fb8a51a3ef65ee0a20b2dfe03f8a0fc1017"} Sep 15 12:02:02.211722 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:02.211582 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" Sep 15 12:02:02.228475 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:02.228422 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" podStartSLOduration=2.228407629 podStartE2EDuration="2.228407629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:02.227117132 +0000 UTC m=+1509.457043669" watchObservedRunningTime="2026-09-15 12:02:02.228407629 +0000 UTC m=+1509.458334194" Sep 15 12:02:04.862654 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:04.862617 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w"] Sep 15 12:02:04.863157 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:04.862864 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" podUID="0bc02592-5c04-4a91-8306-5fe1c540169a" containerName="payload-processing" containerID="cri-o://ca68b7da6e5a5bdd79566626e3b23d0fc4127349b8f3d1ab48f4f59fb7ace71f" gracePeriod=30 Sep 15 12:02:04.870319 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:04.870197 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v"] Sep 15 12:02:04.870470 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:04.870442 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" podUID="25341636-cbaf-4bfd-b040-c7e4bf118e06" containerName="payload-pre-processing" containerID="cri-o://8511668b25b2b9e48a0acf66ea25d61c3144323ab75cb81f3a7213c06cb43dcd" gracePeriod=30 Sep 15 12:02:05.043231 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.043190 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6"] Sep 15 12:02:05.046715 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.046644 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7f9f55-rngb6"] Sep 15 12:02:05.149665 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.149643 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" Sep 15 12:02:05.152411 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.152392 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" Sep 15 12:02:05.159837 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.159809 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rbxc\" (UniqueName: \"kubernetes.io/projected/25341636-cbaf-4bfd-b040-c7e4bf118e06-kube-api-access-6rbxc\") pod \"25341636-cbaf-4bfd-b040-c7e4bf118e06\" (UID: \"25341636-cbaf-4bfd-b040-c7e4bf118e06\") " Sep 15 12:02:05.159951 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.159868 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25341636-cbaf-4bfd-b040-c7e4bf118e06-plugins-config-volume\") pod \"25341636-cbaf-4bfd-b040-c7e4bf118e06\" (UID: \"25341636-cbaf-4bfd-b040-c7e4bf118e06\") " Sep 15 12:02:05.159951 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.159921 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc02592-5c04-4a91-8306-5fe1c540169a-plugins-config-volume\") pod \"0bc02592-5c04-4a91-8306-5fe1c540169a\" (UID: \"0bc02592-5c04-4a91-8306-5fe1c540169a\") " Sep 15 12:02:05.160070 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.159975 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz5vf\" (UniqueName: \"kubernetes.io/projected/0bc02592-5c04-4a91-8306-5fe1c540169a-kube-api-access-pz5vf\") pod \"0bc02592-5c04-4a91-8306-5fe1c540169a\" (UID: \"0bc02592-5c04-4a91-8306-5fe1c540169a\") " Sep 15 12:02:05.160330 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.160309 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25341636-cbaf-4bfd-b040-c7e4bf118e06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25341636-cbaf-4bfd-b040-c7e4bf118e06" (UID: "25341636-cbaf-4bfd-b040-c7e4bf118e06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:02:05.160408 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.160343 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc02592-5c04-4a91-8306-5fe1c540169a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0bc02592-5c04-4a91-8306-5fe1c540169a" (UID: "0bc02592-5c04-4a91-8306-5fe1c540169a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:02:05.161883 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.161855 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25341636-cbaf-4bfd-b040-c7e4bf118e06-kube-api-access-6rbxc" (OuterVolumeSpecName: "kube-api-access-6rbxc") pod "25341636-cbaf-4bfd-b040-c7e4bf118e06" (UID: "25341636-cbaf-4bfd-b040-c7e4bf118e06"). InnerVolumeSpecName "kube-api-access-6rbxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:02:05.161965 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.161940 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc02592-5c04-4a91-8306-5fe1c540169a-kube-api-access-pz5vf" (OuterVolumeSpecName: "kube-api-access-pz5vf") pod "0bc02592-5c04-4a91-8306-5fe1c540169a" (UID: "0bc02592-5c04-4a91-8306-5fe1c540169a"). InnerVolumeSpecName "kube-api-access-pz5vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:02:05.228410 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.228373 2573 generic.go:356] "Generic (PLEG): container finished" podID="0bc02592-5c04-4a91-8306-5fe1c540169a" containerID="ca68b7da6e5a5bdd79566626e3b23d0fc4127349b8f3d1ab48f4f59fb7ace71f" exitCode=0 Sep 15 12:02:05.228578 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.228487 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" Sep 15 12:02:05.229885 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.229856 2573 generic.go:356] "Generic (PLEG): container finished" podID="25341636-cbaf-4bfd-b040-c7e4bf118e06" containerID="8511668b25b2b9e48a0acf66ea25d61c3144323ab75cb81f3a7213c06cb43dcd" exitCode=0 Sep 15 12:02:05.229995 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.229953 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" Sep 15 12:02:05.235090 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.235065 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3" path="/var/lib/kubelet/pods/fe5f0345-ae67-4bbf-ace2-f8a0e3e8b9d3/volumes" Sep 15 12:02:05.235576 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.235558 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" event={"ID":"0bc02592-5c04-4a91-8306-5fe1c540169a","Type":"ContainerDied","Data":"ca68b7da6e5a5bdd79566626e3b23d0fc4127349b8f3d1ab48f4f59fb7ace71f"} Sep 15 12:02:05.235659 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.235587 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w" event={"ID":"0bc02592-5c04-4a91-8306-5fe1c540169a","Type":"ContainerDied","Data":"b3fb13fdbe7bffc0f8950b2285c97965c88f41192de4327a6630525819ed92b6"} Sep 15 12:02:05.235659 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.235609 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" event={"ID":"25341636-cbaf-4bfd-b040-c7e4bf118e06","Type":"ContainerDied","Data":"8511668b25b2b9e48a0acf66ea25d61c3144323ab75cb81f3a7213c06cb43dcd"} Sep 15 12:02:05.235659 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.235631 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v" event={"ID":"25341636-cbaf-4bfd-b040-c7e4bf118e06","Type":"ContainerDied","Data":"b730ce09e2df242c33f7b0d6580efa4162d9ff9c1dbb5bbe77faf2ebde99563f"} Sep 15 12:02:05.235817 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.235658 2573 scope.go:117] "RemoveContainer" containerID="ca68b7da6e5a5bdd79566626e3b23d0fc4127349b8f3d1ab48f4f59fb7ace71f" Sep 15 12:02:05.245629 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.245614 2573 scope.go:117] "RemoveContainer" containerID="ca68b7da6e5a5bdd79566626e3b23d0fc4127349b8f3d1ab48f4f59fb7ace71f" Sep 15 12:02:05.245919 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:02:05.245901 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca68b7da6e5a5bdd79566626e3b23d0fc4127349b8f3d1ab48f4f59fb7ace71f\": container with ID starting with ca68b7da6e5a5bdd79566626e3b23d0fc4127349b8f3d1ab48f4f59fb7ace71f not found: ID does not exist" containerID="ca68b7da6e5a5bdd79566626e3b23d0fc4127349b8f3d1ab48f4f59fb7ace71f" Sep 15 12:02:05.245960 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.245928 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca68b7da6e5a5bdd79566626e3b23d0fc4127349b8f3d1ab48f4f59fb7ace71f"} err="failed to get container status \"ca68b7da6e5a5bdd79566626e3b23d0fc4127349b8f3d1ab48f4f59fb7ace71f\": rpc error: code = NotFound desc = could not find container \"ca68b7da6e5a5bdd79566626e3b23d0fc4127349b8f3d1ab48f4f59fb7ace71f\": container with ID starting with ca68b7da6e5a5bdd79566626e3b23d0fc4127349b8f3d1ab48f4f59fb7ace71f not found: ID does not exist" Sep 15 12:02:05.245960 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.245945 2573 scope.go:117] "RemoveContainer" containerID="8511668b25b2b9e48a0acf66ea25d61c3144323ab75cb81f3a7213c06cb43dcd" Sep 15 12:02:05.256298 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.256153 2573 scope.go:117] "RemoveContainer" containerID="8511668b25b2b9e48a0acf66ea25d61c3144323ab75cb81f3a7213c06cb43dcd" Sep 15 12:02:05.256376 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.256316 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w"] Sep 15 12:02:05.256944 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:02:05.256908 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8511668b25b2b9e48a0acf66ea25d61c3144323ab75cb81f3a7213c06cb43dcd\": container with ID starting with 8511668b25b2b9e48a0acf66ea25d61c3144323ab75cb81f3a7213c06cb43dcd not found: ID does not exist" containerID="8511668b25b2b9e48a0acf66ea25d61c3144323ab75cb81f3a7213c06cb43dcd" Sep 15 12:02:05.257174 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.257136 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8511668b25b2b9e48a0acf66ea25d61c3144323ab75cb81f3a7213c06cb43dcd"} err="failed to get container status \"8511668b25b2b9e48a0acf66ea25d61c3144323ab75cb81f3a7213c06cb43dcd\": rpc error: code = NotFound desc = could not find container \"8511668b25b2b9e48a0acf66ea25d61c3144323ab75cb81f3a7213c06cb43dcd\": container with ID starting with 8511668b25b2b9e48a0acf66ea25d61c3144323ab75cb81f3a7213c06cb43dcd not found: ID does not exist" Sep 15 12:02:05.259274 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.259251 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7f9f55-669854568c-zlx7w"] Sep 15 12:02:05.260776 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.260758 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc02592-5c04-4a91-8306-5fe1c540169a-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:02:05.260846 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.260779 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz5vf\" (UniqueName: \"kubernetes.io/projected/0bc02592-5c04-4a91-8306-5fe1c540169a-kube-api-access-pz5vf\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:02:05.260846 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.260789 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rbxc\" (UniqueName: \"kubernetes.io/projected/25341636-cbaf-4bfd-b040-c7e4bf118e06-kube-api-access-6rbxc\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:02:05.260846 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.260798 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25341636-cbaf-4bfd-b040-c7e4bf118e06-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:02:05.270823 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.270799 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v"] Sep 15 12:02:05.275000 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:05.274978 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7f9f55-57894ddd6b-ghz2v"] Sep 15 12:02:07.216901 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:07.216872 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:02:07.235781 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:07.235751 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bc02592-5c04-4a91-8306-5fe1c540169a" path="/var/lib/kubelet/pods/0bc02592-5c04-4a91-8306-5fe1c540169a/volumes" Sep 15 12:02:07.236228 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:07.236190 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25341636-cbaf-4bfd-b040-c7e4bf118e06" path="/var/lib/kubelet/pods/25341636-cbaf-4bfd-b040-c7e4bf118e06/volumes" Sep 15 12:02:13.219619 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:13.219580 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" Sep 15 12:02:23.848830 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:23.848744 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d2de76-29824560-d974s"] Sep 15 12:02:23.857533 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:23.857505 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d2de76-29824560-d974s"] Sep 15 12:02:23.886359 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:23.886325 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5"] Sep 15 12:02:23.886622 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:23.886573 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" podUID="8d61f56c-a69c-439a-950c-d40dbf997396" containerName="payload-processing" containerID="cri-o://14543f644b23a4dcbd92ee0d266124ff30135472c26a74886babc5fde2ec7190" gracePeriod=30 Sep 15 12:02:23.900430 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:23.900399 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn"] Sep 15 12:02:23.900697 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:23.900647 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" podUID="2a680930-2cc1-41d4-833d-4eca26a6f52a" containerName="payload-pre-processing" containerID="cri-o://3b1868b2d0a1bc76be2c8f9ce7ed5612e19e5444a5ed09db1b7ecde32edcbbcf" gracePeriod=30 Sep 15 12:02:24.172055 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.172010 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" Sep 15 12:02:24.174845 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.174825 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" Sep 15 12:02:24.230966 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.230936 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a680930-2cc1-41d4-833d-4eca26a6f52a-plugins-config-volume\") pod \"2a680930-2cc1-41d4-833d-4eca26a6f52a\" (UID: \"2a680930-2cc1-41d4-833d-4eca26a6f52a\") " Sep 15 12:02:24.230966 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.230971 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q47fx\" (UniqueName: \"kubernetes.io/projected/8d61f56c-a69c-439a-950c-d40dbf997396-kube-api-access-q47fx\") pod \"8d61f56c-a69c-439a-950c-d40dbf997396\" (UID: \"8d61f56c-a69c-439a-950c-d40dbf997396\") " Sep 15 12:02:24.231212 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.231087 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6w7q\" (UniqueName: \"kubernetes.io/projected/2a680930-2cc1-41d4-833d-4eca26a6f52a-kube-api-access-j6w7q\") pod \"2a680930-2cc1-41d4-833d-4eca26a6f52a\" (UID: \"2a680930-2cc1-41d4-833d-4eca26a6f52a\") " Sep 15 12:02:24.231212 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.231116 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d61f56c-a69c-439a-950c-d40dbf997396-plugins-config-volume\") pod \"8d61f56c-a69c-439a-950c-d40dbf997396\" (UID: \"8d61f56c-a69c-439a-950c-d40dbf997396\") " Sep 15 12:02:24.231378 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.231351 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a680930-2cc1-41d4-833d-4eca26a6f52a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a680930-2cc1-41d4-833d-4eca26a6f52a" (UID: "2a680930-2cc1-41d4-833d-4eca26a6f52a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:02:24.231634 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.231602 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d61f56c-a69c-439a-950c-d40dbf997396-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d61f56c-a69c-439a-950c-d40dbf997396" (UID: "8d61f56c-a69c-439a-950c-d40dbf997396"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:02:24.233166 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.233145 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d61f56c-a69c-439a-950c-d40dbf997396-kube-api-access-q47fx" (OuterVolumeSpecName: "kube-api-access-q47fx") pod "8d61f56c-a69c-439a-950c-d40dbf997396" (UID: "8d61f56c-a69c-439a-950c-d40dbf997396"). InnerVolumeSpecName "kube-api-access-q47fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:02:24.233258 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.233170 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a680930-2cc1-41d4-833d-4eca26a6f52a-kube-api-access-j6w7q" (OuterVolumeSpecName: "kube-api-access-j6w7q") pod "2a680930-2cc1-41d4-833d-4eca26a6f52a" (UID: "2a680930-2cc1-41d4-833d-4eca26a6f52a"). InnerVolumeSpecName "kube-api-access-j6w7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:02:24.331809 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.331778 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a680930-2cc1-41d4-833d-4eca26a6f52a-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:02:24.331809 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.331803 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q47fx\" (UniqueName: \"kubernetes.io/projected/8d61f56c-a69c-439a-950c-d40dbf997396-kube-api-access-q47fx\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:02:24.331809 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.331812 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6w7q\" (UniqueName: \"kubernetes.io/projected/2a680930-2cc1-41d4-833d-4eca26a6f52a-kube-api-access-j6w7q\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:02:24.332107 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.331821 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d61f56c-a69c-439a-950c-d40dbf997396-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:02:24.334821 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.334787 2573 generic.go:356] "Generic (PLEG): container finished" podID="2a680930-2cc1-41d4-833d-4eca26a6f52a" containerID="3b1868b2d0a1bc76be2c8f9ce7ed5612e19e5444a5ed09db1b7ecde32edcbbcf" exitCode=0 Sep 15 12:02:24.334945 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.334828 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" Sep 15 12:02:24.334945 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.334872 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" event={"ID":"2a680930-2cc1-41d4-833d-4eca26a6f52a","Type":"ContainerDied","Data":"3b1868b2d0a1bc76be2c8f9ce7ed5612e19e5444a5ed09db1b7ecde32edcbbcf"} Sep 15 12:02:24.334945 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.334926 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn" event={"ID":"2a680930-2cc1-41d4-833d-4eca26a6f52a","Type":"ContainerDied","Data":"a420aeec3a7d0698d37128fdc6f9560aafb6e3b596a85e0e1e5c36040915b2de"} Sep 15 12:02:24.335115 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.334955 2573 scope.go:117] "RemoveContainer" containerID="3b1868b2d0a1bc76be2c8f9ce7ed5612e19e5444a5ed09db1b7ecde32edcbbcf" Sep 15 12:02:24.336269 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.336248 2573 generic.go:356] "Generic (PLEG): container finished" podID="8d61f56c-a69c-439a-950c-d40dbf997396" containerID="14543f644b23a4dcbd92ee0d266124ff30135472c26a74886babc5fde2ec7190" exitCode=0 Sep 15 12:02:24.336410 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.336302 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" Sep 15 12:02:24.336410 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.336331 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" event={"ID":"8d61f56c-a69c-439a-950c-d40dbf997396","Type":"ContainerDied","Data":"14543f644b23a4dcbd92ee0d266124ff30135472c26a74886babc5fde2ec7190"} Sep 15 12:02:24.336410 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.336380 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5" event={"ID":"8d61f56c-a69c-439a-950c-d40dbf997396","Type":"ContainerDied","Data":"74a3b7bc49fef80193a9aa89d5a0c4656f0f3bbd134bbaecebe881ef8fa34b04"} Sep 15 12:02:24.343460 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.343442 2573 scope.go:117] "RemoveContainer" containerID="3b1868b2d0a1bc76be2c8f9ce7ed5612e19e5444a5ed09db1b7ecde32edcbbcf" Sep 15 12:02:24.343722 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:02:24.343702 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1868b2d0a1bc76be2c8f9ce7ed5612e19e5444a5ed09db1b7ecde32edcbbcf\": container with ID starting with 3b1868b2d0a1bc76be2c8f9ce7ed5612e19e5444a5ed09db1b7ecde32edcbbcf not found: ID does not exist" containerID="3b1868b2d0a1bc76be2c8f9ce7ed5612e19e5444a5ed09db1b7ecde32edcbbcf" Sep 15 12:02:24.343792 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.343730 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1868b2d0a1bc76be2c8f9ce7ed5612e19e5444a5ed09db1b7ecde32edcbbcf"} err="failed to get container status \"3b1868b2d0a1bc76be2c8f9ce7ed5612e19e5444a5ed09db1b7ecde32edcbbcf\": rpc error: code = NotFound desc = could not find container \"3b1868b2d0a1bc76be2c8f9ce7ed5612e19e5444a5ed09db1b7ecde32edcbbcf\": container with ID starting with 3b1868b2d0a1bc76be2c8f9ce7ed5612e19e5444a5ed09db1b7ecde32edcbbcf not found: ID does not exist" Sep 15 12:02:24.343792 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.343747 2573 scope.go:117] "RemoveContainer" containerID="14543f644b23a4dcbd92ee0d266124ff30135472c26a74886babc5fde2ec7190" Sep 15 12:02:24.352183 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.352165 2573 scope.go:117] "RemoveContainer" containerID="14543f644b23a4dcbd92ee0d266124ff30135472c26a74886babc5fde2ec7190" Sep 15 12:02:24.352473 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:02:24.352451 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14543f644b23a4dcbd92ee0d266124ff30135472c26a74886babc5fde2ec7190\": container with ID starting with 14543f644b23a4dcbd92ee0d266124ff30135472c26a74886babc5fde2ec7190 not found: ID does not exist" containerID="14543f644b23a4dcbd92ee0d266124ff30135472c26a74886babc5fde2ec7190" Sep 15 12:02:24.352554 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.352479 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14543f644b23a4dcbd92ee0d266124ff30135472c26a74886babc5fde2ec7190"} err="failed to get container status \"14543f644b23a4dcbd92ee0d266124ff30135472c26a74886babc5fde2ec7190\": rpc error: code = NotFound desc = could not find container \"14543f644b23a4dcbd92ee0d266124ff30135472c26a74886babc5fde2ec7190\": container with ID starting with 14543f644b23a4dcbd92ee0d266124ff30135472c26a74886babc5fde2ec7190 not found: ID does not exist" Sep 15 12:02:24.361478 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.361453 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn"] Sep 15 12:02:24.366103 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.366079 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d2de76-58c96699d-cjdbn"] Sep 15 12:02:24.385045 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.380341 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5"] Sep 15 12:02:24.388408 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:24.388387 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d2de76-85bd49cd9-hbck5"] Sep 15 12:02:25.234298 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:25.234259 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a680930-2cc1-41d4-833d-4eca26a6f52a" path="/var/lib/kubelet/pods/2a680930-2cc1-41d4-833d-4eca26a6f52a/volumes" Sep 15 12:02:25.234804 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:25.234791 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d61f56c-a69c-439a-950c-d40dbf997396" path="/var/lib/kubelet/pods/8d61f56c-a69c-439a-950c-d40dbf997396/volumes" Sep 15 12:02:25.235136 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:25.235115 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb66bd79-e33f-4c69-ab81-04e5c008541b" path="/var/lib/kubelet/pods/bb66bd79-e33f-4c69-ab81-04e5c008541b/volumes" Sep 15 12:02:33.703256 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.703221 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl"] Sep 15 12:02:33.703627 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.703577 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc02592-5c04-4a91-8306-5fe1c540169a" containerName="payload-processing" Sep 15 12:02:33.703627 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.703589 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc02592-5c04-4a91-8306-5fe1c540169a" containerName="payload-processing" Sep 15 12:02:33.703627 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.703600 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a680930-2cc1-41d4-833d-4eca26a6f52a" containerName="payload-pre-processing" Sep 15 12:02:33.703627 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.703606 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a680930-2cc1-41d4-833d-4eca26a6f52a" containerName="payload-pre-processing" Sep 15 12:02:33.703627 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.703621 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25341636-cbaf-4bfd-b040-c7e4bf118e06" containerName="payload-pre-processing" Sep 15 12:02:33.703787 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.703630 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="25341636-cbaf-4bfd-b040-c7e4bf118e06" containerName="payload-pre-processing" Sep 15 12:02:33.703787 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.703648 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d61f56c-a69c-439a-950c-d40dbf997396" containerName="payload-processing" Sep 15 12:02:33.703787 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.703653 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d61f56c-a69c-439a-950c-d40dbf997396" containerName="payload-processing" Sep 15 12:02:33.703787 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.703728 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="25341636-cbaf-4bfd-b040-c7e4bf118e06" containerName="payload-pre-processing" Sep 15 12:02:33.703787 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.703735 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d61f56c-a69c-439a-950c-d40dbf997396" containerName="payload-processing" Sep 15 12:02:33.703787 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.703744 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bc02592-5c04-4a91-8306-5fe1c540169a" containerName="payload-processing" Sep 15 12:02:33.703787 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.703751 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a680930-2cc1-41d4-833d-4eca26a6f52a" containerName="payload-pre-processing" Sep 15 12:02:33.708036 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.708003 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.711438 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.711415 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-030981\"/\"kube-root-ca.crt\"" Sep 15 12:02:33.711438 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.711426 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-030981\"/\"ipp-route-030981-kserve-self-signed-certs\"" Sep 15 12:02:33.711632 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.711421 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-030981\"/\"openshift-service-ca.crt\"" Sep 15 12:02:33.711632 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.711421 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-030981\"/\"default-dockercfg-zr6sb\"" Sep 15 12:02:33.716335 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.716307 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl"] Sep 15 12:02:33.819401 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.819360 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-tmp-dir\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.819599 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.819433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-model-cache\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.819599 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.819475 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-dshm\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.819599 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.819558 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8cc\" (UniqueName: \"kubernetes.io/projected/fcf7c68f-0e78-44de-8d53-b38656851f46-kube-api-access-mk8cc\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.819722 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.819633 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf7c68f-0e78-44de-8d53-b38656851f46-tls-certs\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.819722 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.819676 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-home\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.920293 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.920257 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-dshm\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.920484 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.920345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8cc\" (UniqueName: \"kubernetes.io/projected/fcf7c68f-0e78-44de-8d53-b38656851f46-kube-api-access-mk8cc\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.920484 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.920376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf7c68f-0e78-44de-8d53-b38656851f46-tls-certs\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.920484 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.920402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-home\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.920484 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.920434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-tmp-dir\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.920733 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.920491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-model-cache\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.920830 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.920807 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-home\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.920876 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.920819 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-tmp-dir\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.920876 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.920839 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-model-cache\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.922586 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.922567 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-dshm\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.922943 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.922923 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf7c68f-0e78-44de-8d53-b38656851f46-tls-certs\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:33.928614 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:33.928591 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8cc\" (UniqueName: \"kubernetes.io/projected/fcf7c68f-0e78-44de-8d53-b38656851f46-kube-api-access-mk8cc\") pod \"ipp-route-030981-kserve-5b867445b7-rv5wl\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:34.018451 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:34.018418 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:34.293137 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:34.293099 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl"] Sep 15 12:02:34.295801 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:02:34.295774 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcf7c68f_0e78_44de_8d53_b38656851f46.slice/crio-d4f5e7056fbe4d31e5b0eab8251348acba6c0a5bde05fb3badf08d7f0ec56912 WatchSource:0}: Error finding container d4f5e7056fbe4d31e5b0eab8251348acba6c0a5bde05fb3badf08d7f0ec56912: Status 404 returned error can't find the container with id d4f5e7056fbe4d31e5b0eab8251348acba6c0a5bde05fb3badf08d7f0ec56912 Sep 15 12:02:34.389970 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:34.389931 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" event={"ID":"fcf7c68f-0e78-44de-8d53-b38656851f46","Type":"ContainerStarted","Data":"5d54c79f2cf05095322133029c5838a70c1d3a7e02a10facde1ae59bffec0053"} Sep 15 12:02:34.390157 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:34.389979 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" event={"ID":"fcf7c68f-0e78-44de-8d53-b38656851f46","Type":"ContainerStarted","Data":"d4f5e7056fbe4d31e5b0eab8251348acba6c0a5bde05fb3badf08d7f0ec56912"} Sep 15 12:02:34.412270 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:34.412221 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" podStartSLOduration=1.412206209 podStartE2EDuration="1.412206209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:34.409596188 +0000 UTC m=+1541.639522724" watchObservedRunningTime="2026-09-15 12:02:34.412206209 +0000 UTC m=+1541.642132744" Sep 15 12:02:44.019197 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:44.019157 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:44.019585 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:44.019216 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:44.031642 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:44.031616 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:02:44.448799 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:02:44.448712 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:03:06.476201 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.476163 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx"] Sep 15 12:03:06.479911 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.479889 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:06.482400 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.482376 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-c04390-serving-cert\"" Sep 15 12:03:06.494503 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.494477 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx"] Sep 15 12:03:06.603797 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.603761 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:06.603797 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.603800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:06.604004 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.603838 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqxn2\" (UniqueName: \"kubernetes.io/projected/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-kube-api-access-lqxn2\") pod \"maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:06.604004 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.603883 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:06.704608 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.704570 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxn2\" (UniqueName: \"kubernetes.io/projected/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-kube-api-access-lqxn2\") pod \"maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:06.704810 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.704623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:06.704810 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.704694 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:06.704810 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.704712 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:06.705382 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.705354 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:06.707081 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.707054 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:06.707189 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.707114 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:06.712200 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.712180 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxn2\" (UniqueName: \"kubernetes.io/projected/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-kube-api-access-lqxn2\") pod \"maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:06.790342 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:06.790314 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:07.068691 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.068663 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx"] Sep 15 12:03:07.071532 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:03:07.071498 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c15bb61_0d09_40ab_ba4b_e4d38f80af07.slice/crio-d2321301cf547ac0cda4d44ed5957dac717167342de518d65f05d111a3422d56 WatchSource:0}: Error finding container d2321301cf547ac0cda4d44ed5957dac717167342de518d65f05d111a3422d56: Status 404 returned error can't find the container with id d2321301cf547ac0cda4d44ed5957dac717167342de518d65f05d111a3422d56 Sep 15 12:03:07.072071 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.072049 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp"] Sep 15 12:03:07.076964 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.076945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" Sep 15 12:03:07.079433 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.079413 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-c04390-dockercfg-kdr4k\"" Sep 15 12:03:07.079521 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.079418 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-c04390\"" Sep 15 12:03:07.090769 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.090743 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp"] Sep 15 12:03:07.109852 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.109821 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz"] Sep 15 12:03:07.113300 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.113277 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" Sep 15 12:03:07.123269 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.123241 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz"] Sep 15 12:03:07.208851 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.208814 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x6dm\" (UniqueName: \"kubernetes.io/projected/ce6478c1-8d00-4bf0-98bf-66496a9d9df1-kube-api-access-8x6dm\") pod \"payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz\" (UID: \"ce6478c1-8d00-4bf0-98bf-66496a9d9df1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" Sep 15 12:03:07.208851 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.208848 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmkrf\" (UniqueName: \"kubernetes.io/projected/5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce-kube-api-access-gmkrf\") pod \"payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp\" (UID: \"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" Sep 15 12:03:07.209161 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.208951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp\" (UID: \"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" Sep 15 12:03:07.209161 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.209002 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6478c1-8d00-4bf0-98bf-66496a9d9df1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz\" (UID: \"ce6478c1-8d00-4bf0-98bf-66496a9d9df1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" Sep 15 12:03:07.310195 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.310155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6478c1-8d00-4bf0-98bf-66496a9d9df1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz\" (UID: \"ce6478c1-8d00-4bf0-98bf-66496a9d9df1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" Sep 15 12:03:07.310368 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.310217 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x6dm\" (UniqueName: \"kubernetes.io/projected/ce6478c1-8d00-4bf0-98bf-66496a9d9df1-kube-api-access-8x6dm\") pod \"payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz\" (UID: \"ce6478c1-8d00-4bf0-98bf-66496a9d9df1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" Sep 15 12:03:07.310368 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.310265 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmkrf\" (UniqueName: \"kubernetes.io/projected/5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce-kube-api-access-gmkrf\") pod \"payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp\" (UID: \"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" Sep 15 12:03:07.310517 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.310373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp\" (UID: \"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" Sep 15 12:03:07.311060 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.311009 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6478c1-8d00-4bf0-98bf-66496a9d9df1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz\" (UID: \"ce6478c1-8d00-4bf0-98bf-66496a9d9df1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" Sep 15 12:03:07.311168 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.311069 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp\" (UID: \"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" Sep 15 12:03:07.318469 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.318413 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmkrf\" (UniqueName: \"kubernetes.io/projected/5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce-kube-api-access-gmkrf\") pod \"payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp\" (UID: \"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" Sep 15 12:03:07.318728 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.318711 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x6dm\" (UniqueName: \"kubernetes.io/projected/ce6478c1-8d00-4bf0-98bf-66496a9d9df1-kube-api-access-8x6dm\") pod \"payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz\" (UID: \"ce6478c1-8d00-4bf0-98bf-66496a9d9df1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" Sep 15 12:03:07.388685 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.388658 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" Sep 15 12:03:07.424949 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.424921 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" Sep 15 12:03:07.549357 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.549316 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" event={"ID":"7c15bb61-0d09-40ab-ba4b-e4d38f80af07","Type":"ContainerStarted","Data":"d2321301cf547ac0cda4d44ed5957dac717167342de518d65f05d111a3422d56"} Sep 15 12:03:07.661571 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.661547 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp"] Sep 15 12:03:07.662854 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:03:07.662821 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7bcc06_f5a1_4f2c_bcb8_d89890ec32ce.slice/crio-e865ad3ec412317d0b72487b0e42dbdebd5e17a3780c5ceb743e9753a1d5ac84 WatchSource:0}: Error finding container e865ad3ec412317d0b72487b0e42dbdebd5e17a3780c5ceb743e9753a1d5ac84: Status 404 returned error can't find the container with id e865ad3ec412317d0b72487b0e42dbdebd5e17a3780c5ceb743e9753a1d5ac84 Sep 15 12:03:07.702903 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:07.702871 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz"] Sep 15 12:03:07.704692 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:03:07.704665 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6478c1_8d00_4bf0_98bf_66496a9d9df1.slice/crio-f33552c90d0b8af90ef63f1a93a092385833031621848b6fbe99627af35e8429 WatchSource:0}: Error finding container f33552c90d0b8af90ef63f1a93a092385833031621848b6fbe99627af35e8429: Status 404 returned error can't find the container with id f33552c90d0b8af90ef63f1a93a092385833031621848b6fbe99627af35e8429 Sep 15 12:03:08.555661 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:08.555624 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" event={"ID":"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce","Type":"ContainerStarted","Data":"6160e4607e692aab3bd0f94581e62ab9090057d6711fa22f61be9bc3e7dd0d36"} Sep 15 12:03:08.556188 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:08.555683 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" event={"ID":"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce","Type":"ContainerStarted","Data":"e865ad3ec412317d0b72487b0e42dbdebd5e17a3780c5ceb743e9753a1d5ac84"} Sep 15 12:03:08.556188 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:08.555799 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" Sep 15 12:03:08.557299 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:08.557277 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" event={"ID":"ce6478c1-8d00-4bf0-98bf-66496a9d9df1","Type":"ContainerStarted","Data":"36a1654b1347394328f5b12d2829f6f44a7dcf4676d258493e7cfb61fdbea027"} Sep 15 12:03:08.557396 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:08.557320 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" Sep 15 12:03:08.557396 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:08.557336 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" event={"ID":"ce6478c1-8d00-4bf0-98bf-66496a9d9df1","Type":"ContainerStarted","Data":"f33552c90d0b8af90ef63f1a93a092385833031621848b6fbe99627af35e8429"} Sep 15 12:03:08.573625 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:08.573569 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" podStartSLOduration=1.5735536319999999 podStartE2EDuration="1.573553632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:08.572388466 +0000 UTC m=+1575.802315004" watchObservedRunningTime="2026-09-15 12:03:08.573553632 +0000 UTC m=+1575.803480178" Sep 15 12:03:08.590856 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:08.590804 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" podStartSLOduration=1.590787931 podStartE2EDuration="1.590787931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:08.589071134 +0000 UTC m=+1575.818997670" watchObservedRunningTime="2026-09-15 12:03:08.590787931 +0000 UTC m=+1575.820714470" Sep 15 12:03:11.582801 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:11.582759 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" event={"ID":"7c15bb61-0d09-40ab-ba4b-e4d38f80af07","Type":"ContainerStarted","Data":"2bd2711cd2fe00777ca5c31b7693a4028dd6144f678f2847151a6ae8c6e107cf"} Sep 15 12:03:11.583220 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:11.582847 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:11.604671 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:11.604614 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" podStartSLOduration=1.884029473 podStartE2EDuration="5.604599832s" totalImagesPullingTime="3.720570359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:11.602803082 +0000 UTC m=+1578.832729621" watchObservedRunningTime="2026-09-15 12:03:11.604599832 +0000 UTC m=+1578.834526407" Sep 15 12:03:17.593587 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:17.593556 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:19.565192 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:19.565158 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" Sep 15 12:03:19.565662 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:19.565235 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" Sep 15 12:03:35.468084 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.468049 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx"] Sep 15 12:03:35.468492 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.468274 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" podUID="7c15bb61-0d09-40ab-ba4b-e4d38f80af07" containerName="maas-api" containerID="cri-o://2bd2711cd2fe00777ca5c31b7693a4028dd6144f678f2847151a6ae8c6e107cf" gracePeriod=30 Sep 15 12:03:35.511152 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.511119 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz"] Sep 15 12:03:35.511388 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.511340 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" podUID="ce6478c1-8d00-4bf0-98bf-66496a9d9df1" containerName="payload-processing" containerID="cri-o://36a1654b1347394328f5b12d2829f6f44a7dcf4676d258493e7cfb61fdbea027" gracePeriod=30 Sep 15 12:03:35.520378 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.520345 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp"] Sep 15 12:03:35.521085 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.520822 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" podUID="5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce" containerName="payload-pre-processing" containerID="cri-o://6160e4607e692aab3bd0f94581e62ab9090057d6711fa22f61be9bc3e7dd0d36" gracePeriod=30 Sep 15 12:03:35.708381 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.708345 2573 generic.go:356] "Generic (PLEG): container finished" podID="ce6478c1-8d00-4bf0-98bf-66496a9d9df1" containerID="36a1654b1347394328f5b12d2829f6f44a7dcf4676d258493e7cfb61fdbea027" exitCode=0 Sep 15 12:03:35.708524 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.708393 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" event={"ID":"ce6478c1-8d00-4bf0-98bf-66496a9d9df1","Type":"ContainerDied","Data":"36a1654b1347394328f5b12d2829f6f44a7dcf4676d258493e7cfb61fdbea027"} Sep 15 12:03:35.710538 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.710512 2573 generic.go:356] "Generic (PLEG): container finished" podID="7c15bb61-0d09-40ab-ba4b-e4d38f80af07" containerID="2bd2711cd2fe00777ca5c31b7693a4028dd6144f678f2847151a6ae8c6e107cf" exitCode=0 Sep 15 12:03:35.710601 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.710539 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" event={"ID":"7c15bb61-0d09-40ab-ba4b-e4d38f80af07","Type":"ContainerDied","Data":"2bd2711cd2fe00777ca5c31b7693a4028dd6144f678f2847151a6ae8c6e107cf"} Sep 15 12:03:35.712249 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.712228 2573 generic.go:356] "Generic (PLEG): container finished" podID="5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce" containerID="6160e4607e692aab3bd0f94581e62ab9090057d6711fa22f61be9bc3e7dd0d36" exitCode=0 Sep 15 12:03:35.712325 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.712261 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" event={"ID":"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce","Type":"ContainerDied","Data":"6160e4607e692aab3bd0f94581e62ab9090057d6711fa22f61be9bc3e7dd0d36"} Sep 15 12:03:35.807245 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.807220 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" Sep 15 12:03:35.810260 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.810241 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:35.813588 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.813570 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" Sep 15 12:03:35.891221 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.891188 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-openshift-service-ca\") pod \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " Sep 15 12:03:35.891221 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.891223 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmkrf\" (UniqueName: \"kubernetes.io/projected/5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce-kube-api-access-gmkrf\") pod \"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce\" (UID: \"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce\") " Sep 15 12:03:35.891460 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.891245 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-metrics-certs\") pod \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " Sep 15 12:03:35.891460 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.891311 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x6dm\" (UniqueName: \"kubernetes.io/projected/ce6478c1-8d00-4bf0-98bf-66496a9d9df1-kube-api-access-8x6dm\") pod \"ce6478c1-8d00-4bf0-98bf-66496a9d9df1\" (UID: \"ce6478c1-8d00-4bf0-98bf-66496a9d9df1\") " Sep 15 12:03:35.891460 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.891336 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce-plugins-config-volume\") pod \"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce\" (UID: \"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce\") " Sep 15 12:03:35.891460 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.891361 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6478c1-8d00-4bf0-98bf-66496a9d9df1-plugins-config-volume\") pod \"ce6478c1-8d00-4bf0-98bf-66496a9d9df1\" (UID: \"ce6478c1-8d00-4bf0-98bf-66496a9d9df1\") " Sep 15 12:03:35.891460 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.891438 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqxn2\" (UniqueName: \"kubernetes.io/projected/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-kube-api-access-lqxn2\") pod \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " Sep 15 12:03:35.891460 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.891461 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-maas-api-tls\") pod \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\" (UID: \"7c15bb61-0d09-40ab-ba4b-e4d38f80af07\") " Sep 15 12:03:35.891779 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.891620 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c15bb61-0d09-40ab-ba4b-e4d38f80af07" (UID: "7c15bb61-0d09-40ab-ba4b-e4d38f80af07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:03:35.891779 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.891721 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce" (UID: "5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:03:35.891779 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.891740 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-openshift-service-ca\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:03:35.892066 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.892011 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6478c1-8d00-4bf0-98bf-66496a9d9df1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce6478c1-8d00-4bf0-98bf-66496a9d9df1" (UID: "ce6478c1-8d00-4bf0-98bf-66496a9d9df1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:03:35.893736 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.893708 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce-kube-api-access-gmkrf" (OuterVolumeSpecName: "kube-api-access-gmkrf") pod "5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce" (UID: "5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce"). InnerVolumeSpecName "kube-api-access-gmkrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:03:35.893736 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.893738 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6478c1-8d00-4bf0-98bf-66496a9d9df1-kube-api-access-8x6dm" (OuterVolumeSpecName: "kube-api-access-8x6dm") pod "ce6478c1-8d00-4bf0-98bf-66496a9d9df1" (UID: "ce6478c1-8d00-4bf0-98bf-66496a9d9df1"). InnerVolumeSpecName "kube-api-access-8x6dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:03:35.893899 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.893867 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7c15bb61-0d09-40ab-ba4b-e4d38f80af07" (UID: "7c15bb61-0d09-40ab-ba4b-e4d38f80af07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:03:35.894137 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.894115 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c15bb61-0d09-40ab-ba4b-e4d38f80af07" (UID: "7c15bb61-0d09-40ab-ba4b-e4d38f80af07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:03:35.894239 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.894209 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-kube-api-access-lqxn2" (OuterVolumeSpecName: "kube-api-access-lqxn2") pod "7c15bb61-0d09-40ab-ba4b-e4d38f80af07" (UID: "7c15bb61-0d09-40ab-ba4b-e4d38f80af07"). InnerVolumeSpecName "kube-api-access-lqxn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:03:35.993170 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.993063 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x6dm\" (UniqueName: \"kubernetes.io/projected/ce6478c1-8d00-4bf0-98bf-66496a9d9df1-kube-api-access-8x6dm\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:03:35.993170 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.993100 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:03:35.993170 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.993114 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6478c1-8d00-4bf0-98bf-66496a9d9df1-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:03:35.993170 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.993129 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqxn2\" (UniqueName: \"kubernetes.io/projected/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-kube-api-access-lqxn2\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:03:35.993170 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.993143 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-maas-api-tls\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:03:35.993170 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.993156 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmkrf\" (UniqueName: \"kubernetes.io/projected/5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce-kube-api-access-gmkrf\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:03:35.993170 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:35.993169 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c15bb61-0d09-40ab-ba4b-e4d38f80af07-metrics-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:03:36.718094 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.718065 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" Sep 15 12:03:36.718094 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.718075 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz" event={"ID":"ce6478c1-8d00-4bf0-98bf-66496a9d9df1","Type":"ContainerDied","Data":"f33552c90d0b8af90ef63f1a93a092385833031621848b6fbe99627af35e8429"} Sep 15 12:03:36.718600 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.718136 2573 scope.go:117] "RemoveContainer" containerID="36a1654b1347394328f5b12d2829f6f44a7dcf4676d258493e7cfb61fdbea027" Sep 15 12:03:36.719704 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.719681 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" Sep 15 12:03:36.719704 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.719680 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx" event={"ID":"7c15bb61-0d09-40ab-ba4b-e4d38f80af07","Type":"ContainerDied","Data":"d2321301cf547ac0cda4d44ed5957dac717167342de518d65f05d111a3422d56"} Sep 15 12:03:36.721096 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.721061 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" event={"ID":"5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce","Type":"ContainerDied","Data":"e865ad3ec412317d0b72487b0e42dbdebd5e17a3780c5ceb743e9753a1d5ac84"} Sep 15 12:03:36.721221 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.721074 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp" Sep 15 12:03:36.728350 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.728330 2573 scope.go:117] "RemoveContainer" containerID="2bd2711cd2fe00777ca5c31b7693a4028dd6144f678f2847151a6ae8c6e107cf" Sep 15 12:03:36.737638 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.737617 2573 scope.go:117] "RemoveContainer" containerID="6160e4607e692aab3bd0f94581e62ab9090057d6711fa22f61be9bc3e7dd0d36" Sep 15 12:03:36.744971 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.744947 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz"] Sep 15 12:03:36.748452 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.748430 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c04390-6585674cb9-gsdnz"] Sep 15 12:03:36.760322 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.760300 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp"] Sep 15 12:03:36.764495 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.764471 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c04390-56d6c8b7c4-jhbkp"] Sep 15 12:03:36.774352 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.774331 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx"] Sep 15 12:03:36.777830 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:36.777806 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c04390-974cb8ff5-zx7zx"] Sep 15 12:03:37.234243 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:37.234210 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce" path="/var/lib/kubelet/pods/5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce/volumes" Sep 15 12:03:37.234613 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:37.234598 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c15bb61-0d09-40ab-ba4b-e4d38f80af07" path="/var/lib/kubelet/pods/7c15bb61-0d09-40ab-ba4b-e4d38f80af07/volumes" Sep 15 12:03:37.234945 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:03:37.234932 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce6478c1-8d00-4bf0-98bf-66496a9d9df1" path="/var/lib/kubelet/pods/ce6478c1-8d00-4bf0-98bf-66496a9d9df1/volumes" Sep 15 12:04:08.450299 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:08.450265 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl"] Sep 15 12:04:23.469396 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.469332 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" podUID="fcf7c68f-0e78-44de-8d53-b38656851f46" containerName="main" containerID="cri-o://5d54c79f2cf05095322133029c5838a70c1d3a7e02a10facde1ae59bffec0053" gracePeriod=45 Sep 15 12:04:23.714013 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.713986 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:04:23.796725 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.796695 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-tmp-dir\") pod \"fcf7c68f-0e78-44de-8d53-b38656851f46\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " Sep 15 12:04:23.796896 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.796774 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-model-cache\") pod \"fcf7c68f-0e78-44de-8d53-b38656851f46\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " Sep 15 12:04:23.796896 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.796806 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-dshm\") pod \"fcf7c68f-0e78-44de-8d53-b38656851f46\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " Sep 15 12:04:23.796896 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.796852 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk8cc\" (UniqueName: \"kubernetes.io/projected/fcf7c68f-0e78-44de-8d53-b38656851f46-kube-api-access-mk8cc\") pod \"fcf7c68f-0e78-44de-8d53-b38656851f46\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " Sep 15 12:04:23.796896 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.796873 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf7c68f-0e78-44de-8d53-b38656851f46-tls-certs\") pod \"fcf7c68f-0e78-44de-8d53-b38656851f46\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " Sep 15 12:04:23.796896 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.796888 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-home\") pod \"fcf7c68f-0e78-44de-8d53-b38656851f46\" (UID: \"fcf7c68f-0e78-44de-8d53-b38656851f46\") " Sep 15 12:04:23.797129 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.796937 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fcf7c68f-0e78-44de-8d53-b38656851f46" (UID: "fcf7c68f-0e78-44de-8d53-b38656851f46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:04:23.797129 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.797002 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-model-cache" (OuterVolumeSpecName: "model-cache") pod "fcf7c68f-0e78-44de-8d53-b38656851f46" (UID: "fcf7c68f-0e78-44de-8d53-b38656851f46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:04:23.797209 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.797153 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-tmp-dir\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:04:23.797209 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.797175 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-model-cache\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:04:23.797209 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.797189 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-home" (OuterVolumeSpecName: "home") pod "fcf7c68f-0e78-44de-8d53-b38656851f46" (UID: "fcf7c68f-0e78-44de-8d53-b38656851f46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:04:23.799129 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.799098 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-dshm" (OuterVolumeSpecName: "dshm") pod "fcf7c68f-0e78-44de-8d53-b38656851f46" (UID: "fcf7c68f-0e78-44de-8d53-b38656851f46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:04:23.799238 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.799192 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf7c68f-0e78-44de-8d53-b38656851f46-kube-api-access-mk8cc" (OuterVolumeSpecName: "kube-api-access-mk8cc") pod "fcf7c68f-0e78-44de-8d53-b38656851f46" (UID: "fcf7c68f-0e78-44de-8d53-b38656851f46"). InnerVolumeSpecName "kube-api-access-mk8cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:04:23.799238 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.799208 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf7c68f-0e78-44de-8d53-b38656851f46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcf7c68f-0e78-44de-8d53-b38656851f46" (UID: "fcf7c68f-0e78-44de-8d53-b38656851f46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:04:23.898326 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.898286 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-dshm\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:04:23.898326 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.898322 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk8cc\" (UniqueName: \"kubernetes.io/projected/fcf7c68f-0e78-44de-8d53-b38656851f46-kube-api-access-mk8cc\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:04:23.898326 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.898333 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf7c68f-0e78-44de-8d53-b38656851f46-tls-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:04:23.898564 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.898343 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf7c68f-0e78-44de-8d53-b38656851f46-home\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:04:23.961197 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.961167 2573 generic.go:356] "Generic (PLEG): container finished" podID="fcf7c68f-0e78-44de-8d53-b38656851f46" containerID="5d54c79f2cf05095322133029c5838a70c1d3a7e02a10facde1ae59bffec0053" exitCode=0 Sep 15 12:04:23.961374 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.961203 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" event={"ID":"fcf7c68f-0e78-44de-8d53-b38656851f46","Type":"ContainerDied","Data":"5d54c79f2cf05095322133029c5838a70c1d3a7e02a10facde1ae59bffec0053"} Sep 15 12:04:23.961374 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.961218 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" Sep 15 12:04:23.961374 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.961235 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl" event={"ID":"fcf7c68f-0e78-44de-8d53-b38656851f46","Type":"ContainerDied","Data":"d4f5e7056fbe4d31e5b0eab8251348acba6c0a5bde05fb3badf08d7f0ec56912"} Sep 15 12:04:23.961374 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.961253 2573 scope.go:117] "RemoveContainer" containerID="5d54c79f2cf05095322133029c5838a70c1d3a7e02a10facde1ae59bffec0053" Sep 15 12:04:23.970423 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.970403 2573 scope.go:117] "RemoveContainer" containerID="5d54c79f2cf05095322133029c5838a70c1d3a7e02a10facde1ae59bffec0053" Sep 15 12:04:23.970707 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:04:23.970689 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d54c79f2cf05095322133029c5838a70c1d3a7e02a10facde1ae59bffec0053\": container with ID starting with 5d54c79f2cf05095322133029c5838a70c1d3a7e02a10facde1ae59bffec0053 not found: ID does not exist" containerID="5d54c79f2cf05095322133029c5838a70c1d3a7e02a10facde1ae59bffec0053" Sep 15 12:04:23.970772 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.970714 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d54c79f2cf05095322133029c5838a70c1d3a7e02a10facde1ae59bffec0053"} err="failed to get container status \"5d54c79f2cf05095322133029c5838a70c1d3a7e02a10facde1ae59bffec0053\": rpc error: code = NotFound desc = could not find container \"5d54c79f2cf05095322133029c5838a70c1d3a7e02a10facde1ae59bffec0053\": container with ID starting with 5d54c79f2cf05095322133029c5838a70c1d3a7e02a10facde1ae59bffec0053 not found: ID does not exist" Sep 15 12:04:23.983829 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.982818 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl"] Sep 15 12:04:23.987784 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:23.987732 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-030981/ipp-route-030981-kserve-5b867445b7-rv5wl"] Sep 15 12:04:25.234626 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:04:25.234587 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcf7c68f-0e78-44de-8d53-b38656851f46" path="/var/lib/kubelet/pods/fcf7c68f-0e78-44de-8d53-b38656851f46/volumes" Sep 15 12:05:03.189059 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.189007 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc"] Sep 15 12:05:03.189676 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.189280 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" podUID="7ba9e261-7244-404f-9a2a-fcb7d615858a" containerName="maas-api" containerID="cri-o://a5356f49017ba1b0c5b10df450fc7afe5a7224b89794f13591eb8ad12dfc6674" gracePeriod=30 Sep 15 12:05:03.257498 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.257466 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57"] Sep 15 12:05:03.257711 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.257686 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" podUID="d6298a16-934c-4717-81e1-78763106cf27" containerName="payload-pre-processing" containerID="cri-o://7345b6189a5474ed0ee4a3ec451be3dc0f4ac6ff60546e941252a15f32dbffee" gracePeriod=30 Sep 15 12:05:03.496374 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.496349 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:05:03.532097 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.532070 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" Sep 15 12:05:03.552319 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.552283 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfz5d\" (UniqueName: \"kubernetes.io/projected/7ba9e261-7244-404f-9a2a-fcb7d615858a-kube-api-access-cfz5d\") pod \"7ba9e261-7244-404f-9a2a-fcb7d615858a\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " Sep 15 12:05:03.552483 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.552341 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ba9e261-7244-404f-9a2a-fcb7d615858a-maas-api-tls\") pod \"7ba9e261-7244-404f-9a2a-fcb7d615858a\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " Sep 15 12:05:03.552483 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.552369 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba9e261-7244-404f-9a2a-fcb7d615858a-openshift-service-ca\") pod \"7ba9e261-7244-404f-9a2a-fcb7d615858a\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " Sep 15 12:05:03.552483 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.552395 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khmdh\" (UniqueName: \"kubernetes.io/projected/d6298a16-934c-4717-81e1-78763106cf27-kube-api-access-khmdh\") pod \"d6298a16-934c-4717-81e1-78763106cf27\" (UID: \"d6298a16-934c-4717-81e1-78763106cf27\") " Sep 15 12:05:03.552483 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.552455 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba9e261-7244-404f-9a2a-fcb7d615858a-metrics-certs\") pod \"7ba9e261-7244-404f-9a2a-fcb7d615858a\" (UID: \"7ba9e261-7244-404f-9a2a-fcb7d615858a\") " Sep 15 12:05:03.552708 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.552505 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6298a16-934c-4717-81e1-78763106cf27-plugins-config-volume\") pod \"d6298a16-934c-4717-81e1-78763106cf27\" (UID: \"d6298a16-934c-4717-81e1-78763106cf27\") " Sep 15 12:05:03.552765 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.552738 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba9e261-7244-404f-9a2a-fcb7d615858a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ba9e261-7244-404f-9a2a-fcb7d615858a" (UID: "7ba9e261-7244-404f-9a2a-fcb7d615858a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:05:03.553648 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.553130 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6298a16-934c-4717-81e1-78763106cf27-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d6298a16-934c-4717-81e1-78763106cf27" (UID: "d6298a16-934c-4717-81e1-78763106cf27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:05:03.555096 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.555063 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba9e261-7244-404f-9a2a-fcb7d615858a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ba9e261-7244-404f-9a2a-fcb7d615858a" (UID: "7ba9e261-7244-404f-9a2a-fcb7d615858a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:05:03.555193 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.555171 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6298a16-934c-4717-81e1-78763106cf27-kube-api-access-khmdh" (OuterVolumeSpecName: "kube-api-access-khmdh") pod "d6298a16-934c-4717-81e1-78763106cf27" (UID: "d6298a16-934c-4717-81e1-78763106cf27"). InnerVolumeSpecName "kube-api-access-khmdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:05:03.555250 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.555231 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba9e261-7244-404f-9a2a-fcb7d615858a-kube-api-access-cfz5d" (OuterVolumeSpecName: "kube-api-access-cfz5d") pod "7ba9e261-7244-404f-9a2a-fcb7d615858a" (UID: "7ba9e261-7244-404f-9a2a-fcb7d615858a"). InnerVolumeSpecName "kube-api-access-cfz5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:05:03.555619 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.555597 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba9e261-7244-404f-9a2a-fcb7d615858a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ba9e261-7244-404f-9a2a-fcb7d615858a" (UID: "7ba9e261-7244-404f-9a2a-fcb7d615858a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:05:03.653517 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.653486 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba9e261-7244-404f-9a2a-fcb7d615858a-metrics-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:03.653517 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.653515 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6298a16-934c-4717-81e1-78763106cf27-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:03.653715 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.653531 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfz5d\" (UniqueName: \"kubernetes.io/projected/7ba9e261-7244-404f-9a2a-fcb7d615858a-kube-api-access-cfz5d\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:03.653715 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.653540 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ba9e261-7244-404f-9a2a-fcb7d615858a-maas-api-tls\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:03.653715 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.653549 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba9e261-7244-404f-9a2a-fcb7d615858a-openshift-service-ca\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:03.653715 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:03.653557 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khmdh\" (UniqueName: \"kubernetes.io/projected/d6298a16-934c-4717-81e1-78763106cf27-kube-api-access-khmdh\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:04.159772 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.159741 2573 generic.go:356] "Generic (PLEG): container finished" podID="7ba9e261-7244-404f-9a2a-fcb7d615858a" containerID="a5356f49017ba1b0c5b10df450fc7afe5a7224b89794f13591eb8ad12dfc6674" exitCode=0 Sep 15 12:05:04.159972 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.159801 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" Sep 15 12:05:04.159972 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.159823 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" event={"ID":"7ba9e261-7244-404f-9a2a-fcb7d615858a","Type":"ContainerDied","Data":"a5356f49017ba1b0c5b10df450fc7afe5a7224b89794f13591eb8ad12dfc6674"} Sep 15 12:05:04.159972 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.159901 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc" event={"ID":"7ba9e261-7244-404f-9a2a-fcb7d615858a","Type":"ContainerDied","Data":"b85aa6949169b6189beb044ab755342ef8d0a83c5c346d9a05139099c0a4ff3d"} Sep 15 12:05:04.159972 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.159922 2573 scope.go:117] "RemoveContainer" containerID="a5356f49017ba1b0c5b10df450fc7afe5a7224b89794f13591eb8ad12dfc6674" Sep 15 12:05:04.161359 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.161331 2573 generic.go:356] "Generic (PLEG): container finished" podID="d6298a16-934c-4717-81e1-78763106cf27" containerID="7345b6189a5474ed0ee4a3ec451be3dc0f4ac6ff60546e941252a15f32dbffee" exitCode=0 Sep 15 12:05:04.161476 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.161363 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" event={"ID":"d6298a16-934c-4717-81e1-78763106cf27","Type":"ContainerDied","Data":"7345b6189a5474ed0ee4a3ec451be3dc0f4ac6ff60546e941252a15f32dbffee"} Sep 15 12:05:04.161476 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.161376 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" Sep 15 12:05:04.161476 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.161395 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57" event={"ID":"d6298a16-934c-4717-81e1-78763106cf27","Type":"ContainerDied","Data":"1076f47cf3cf60257f094422d0513fb8a51a3ef65ee0a20b2dfe03f8a0fc1017"} Sep 15 12:05:04.170166 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.170151 2573 scope.go:117] "RemoveContainer" containerID="a5356f49017ba1b0c5b10df450fc7afe5a7224b89794f13591eb8ad12dfc6674" Sep 15 12:05:04.170461 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:05:04.170434 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5356f49017ba1b0c5b10df450fc7afe5a7224b89794f13591eb8ad12dfc6674\": container with ID starting with a5356f49017ba1b0c5b10df450fc7afe5a7224b89794f13591eb8ad12dfc6674 not found: ID does not exist" containerID="a5356f49017ba1b0c5b10df450fc7afe5a7224b89794f13591eb8ad12dfc6674" Sep 15 12:05:04.170562 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.170467 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5356f49017ba1b0c5b10df450fc7afe5a7224b89794f13591eb8ad12dfc6674"} err="failed to get container status \"a5356f49017ba1b0c5b10df450fc7afe5a7224b89794f13591eb8ad12dfc6674\": rpc error: code = NotFound desc = could not find container \"a5356f49017ba1b0c5b10df450fc7afe5a7224b89794f13591eb8ad12dfc6674\": container with ID starting with a5356f49017ba1b0c5b10df450fc7afe5a7224b89794f13591eb8ad12dfc6674 not found: ID does not exist" Sep 15 12:05:04.170562 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.170484 2573 scope.go:117] "RemoveContainer" containerID="7345b6189a5474ed0ee4a3ec451be3dc0f4ac6ff60546e941252a15f32dbffee" Sep 15 12:05:04.178017 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.178000 2573 scope.go:117] "RemoveContainer" containerID="7345b6189a5474ed0ee4a3ec451be3dc0f4ac6ff60546e941252a15f32dbffee" Sep 15 12:05:04.178292 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:05:04.178275 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7345b6189a5474ed0ee4a3ec451be3dc0f4ac6ff60546e941252a15f32dbffee\": container with ID starting with 7345b6189a5474ed0ee4a3ec451be3dc0f4ac6ff60546e941252a15f32dbffee not found: ID does not exist" containerID="7345b6189a5474ed0ee4a3ec451be3dc0f4ac6ff60546e941252a15f32dbffee" Sep 15 12:05:04.178351 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.178299 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7345b6189a5474ed0ee4a3ec451be3dc0f4ac6ff60546e941252a15f32dbffee"} err="failed to get container status \"7345b6189a5474ed0ee4a3ec451be3dc0f4ac6ff60546e941252a15f32dbffee\": rpc error: code = NotFound desc = could not find container \"7345b6189a5474ed0ee4a3ec451be3dc0f4ac6ff60546e941252a15f32dbffee\": container with ID starting with 7345b6189a5474ed0ee4a3ec451be3dc0f4ac6ff60546e941252a15f32dbffee not found: ID does not exist" Sep 15 12:05:04.181873 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.181852 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc"] Sep 15 12:05:04.184100 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.184077 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0249d8-7f6f96b74c-d2bcc"] Sep 15 12:05:04.197077 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.197055 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57"] Sep 15 12:05:04.203333 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:04.203310 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0249d8-7b55b995f8-9cv57"] Sep 15 12:05:05.234536 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:05.234507 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ba9e261-7244-404f-9a2a-fcb7d615858a" path="/var/lib/kubelet/pods/7ba9e261-7244-404f-9a2a-fcb7d615858a/volumes" Sep 15 12:05:05.234896 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:05.234865 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6298a16-934c-4717-81e1-78763106cf27" path="/var/lib/kubelet/pods/d6298a16-934c-4717-81e1-78763106cf27/volumes" Sep 15 12:05:10.660867 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.660829 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb"] Sep 15 12:05:10.661302 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661202 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce" containerName="payload-pre-processing" Sep 15 12:05:10.661302 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661215 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce" containerName="payload-pre-processing" Sep 15 12:05:10.661302 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661224 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce6478c1-8d00-4bf0-98bf-66496a9d9df1" containerName="payload-processing" Sep 15 12:05:10.661302 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661231 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce6478c1-8d00-4bf0-98bf-66496a9d9df1" containerName="payload-processing" Sep 15 12:05:10.661302 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661240 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c15bb61-0d09-40ab-ba4b-e4d38f80af07" containerName="maas-api" Sep 15 12:05:10.661302 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661246 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c15bb61-0d09-40ab-ba4b-e4d38f80af07" containerName="maas-api" Sep 15 12:05:10.661302 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661256 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba9e261-7244-404f-9a2a-fcb7d615858a" containerName="maas-api" Sep 15 12:05:10.661302 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661261 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba9e261-7244-404f-9a2a-fcb7d615858a" containerName="maas-api" Sep 15 12:05:10.661302 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661267 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6298a16-934c-4717-81e1-78763106cf27" containerName="payload-pre-processing" Sep 15 12:05:10.661302 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661273 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6298a16-934c-4717-81e1-78763106cf27" containerName="payload-pre-processing" Sep 15 12:05:10.661302 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661282 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf7c68f-0e78-44de-8d53-b38656851f46" containerName="main" Sep 15 12:05:10.661302 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661287 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf7c68f-0e78-44de-8d53-b38656851f46" containerName="main" Sep 15 12:05:10.661677 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661334 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f7bcc06-f5a1-4f2c-bcb8-d89890ec32ce" containerName="payload-pre-processing" Sep 15 12:05:10.661677 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661342 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce6478c1-8d00-4bf0-98bf-66496a9d9df1" containerName="payload-processing" Sep 15 12:05:10.661677 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661348 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c15bb61-0d09-40ab-ba4b-e4d38f80af07" containerName="maas-api" Sep 15 12:05:10.661677 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661355 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf7c68f-0e78-44de-8d53-b38656851f46" containerName="main" Sep 15 12:05:10.661677 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661361 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ba9e261-7244-404f-9a2a-fcb7d615858a" containerName="maas-api" Sep 15 12:05:10.661677 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.661366 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6298a16-934c-4717-81e1-78763106cf27" containerName="payload-pre-processing" Sep 15 12:05:10.665761 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.665743 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb" Sep 15 12:05:10.669208 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.669185 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wjx8x\"" Sep 15 12:05:10.669377 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.669332 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:05:10.675515 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.675490 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb"] Sep 15 12:05:10.711778 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.711751 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f312de-7d08-438d-9650-48c90596e18a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb\" (UID: \"c3f312de-7d08-438d-9650-48c90596e18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb" Sep 15 12:05:10.813035 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.813001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f312de-7d08-438d-9650-48c90596e18a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb\" (UID: \"c3f312de-7d08-438d-9650-48c90596e18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb" Sep 15 12:05:10.813636 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.813610 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f312de-7d08-438d-9650-48c90596e18a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb\" (UID: \"c3f312de-7d08-438d-9650-48c90596e18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb" Sep 15 12:05:10.976833 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:10.976745 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb" Sep 15 12:05:11.253717 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:11.253689 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb"] Sep 15 12:05:11.257533 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:05:11.257504 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f312de_7d08_438d_9650_48c90596e18a.slice/crio-51e4fd13573ecb6f10cb1b0634b658b88f80c6cf42134dc95136547e938d6a1b WatchSource:0}: Error finding container 51e4fd13573ecb6f10cb1b0634b658b88f80c6cf42134dc95136547e938d6a1b: Status 404 returned error can't find the container with id 51e4fd13573ecb6f10cb1b0634b658b88f80c6cf42134dc95136547e938d6a1b Sep 15 12:05:12.202950 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:12.202916 2573 generic.go:356] "Generic (PLEG): container finished" podID="c3f312de-7d08-438d-9650-48c90596e18a" containerID="71f1d0130fd2492ec26c7c6cb3661078edba5a51743f7687b1e2467c015faf2e" exitCode=0 Sep 15 12:05:12.203329 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:12.202985 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb" event={"ID":"c3f312de-7d08-438d-9650-48c90596e18a","Type":"ContainerDied","Data":"71f1d0130fd2492ec26c7c6cb3661078edba5a51743f7687b1e2467c015faf2e"} Sep 15 12:05:12.203329 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:12.203047 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb" event={"ID":"c3f312de-7d08-438d-9650-48c90596e18a","Type":"ContainerStarted","Data":"51e4fd13573ecb6f10cb1b0634b658b88f80c6cf42134dc95136547e938d6a1b"} Sep 15 12:05:13.337318 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:13.337293 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb" Sep 15 12:05:13.434804 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:13.434773 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f312de-7d08-438d-9650-48c90596e18a-maas-api-service-ca\") pod \"c3f312de-7d08-438d-9650-48c90596e18a\" (UID: \"c3f312de-7d08-438d-9650-48c90596e18a\") " Sep 15 12:05:13.435140 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:13.435118 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f312de-7d08-438d-9650-48c90596e18a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3f312de-7d08-438d-9650-48c90596e18a" (UID: "c3f312de-7d08-438d-9650-48c90596e18a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:05:13.535581 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:13.535550 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f312de-7d08-438d-9650-48c90596e18a-maas-api-service-ca\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:14.214638 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:14.214598 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb" event={"ID":"c3f312de-7d08-438d-9650-48c90596e18a","Type":"ContainerDied","Data":"51e4fd13573ecb6f10cb1b0634b658b88f80c6cf42134dc95136547e938d6a1b"} Sep 15 12:05:14.214819 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:14.214680 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e4fd13573ecb6f10cb1b0634b658b88f80c6cf42134dc95136547e938d6a1b" Sep 15 12:05:14.214819 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:14.214618 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb" Sep 15 12:05:21.236740 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.236661 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-42b26e-29824560-ggjdd"] Sep 15 12:05:21.240206 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.240182 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-42b26e-29824560-ggjdd"] Sep 15 12:05:21.270357 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.270322 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd"] Sep 15 12:05:21.270636 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.270591 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" podUID="9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132" containerName="payload-processing" containerID="cri-o://1a65c18aefc82b479429d770bf35f391cd2ae86d45ca41dd4b39bb1e51157fb7" gracePeriod=30 Sep 15 12:05:21.280615 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.280587 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4"] Sep 15 12:05:21.281057 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.281011 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" podUID="a61f0d11-698f-4f14-b46c-2a43e73e571b" containerName="payload-pre-processing" containerID="cri-o://94ed4e8f9e666bf09b797dd3492fc214d0388e215a29bebb1bcb27ea6386692d" gracePeriod=30 Sep 15 12:05:21.451575 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.451542 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb"] Sep 15 12:05:21.456217 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.456148 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-42b26e-zk9jb"] Sep 15 12:05:21.523141 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.523114 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" Sep 15 12:05:21.554122 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.554093 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" Sep 15 12:05:21.606466 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.606432 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132-plugins-config-volume\") pod \"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132\" (UID: \"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132\") " Sep 15 12:05:21.606466 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.606468 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nq8z\" (UniqueName: \"kubernetes.io/projected/a61f0d11-698f-4f14-b46c-2a43e73e571b-kube-api-access-6nq8z\") pod \"a61f0d11-698f-4f14-b46c-2a43e73e571b\" (UID: \"a61f0d11-698f-4f14-b46c-2a43e73e571b\") " Sep 15 12:05:21.606700 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.606516 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61f0d11-698f-4f14-b46c-2a43e73e571b-plugins-config-volume\") pod \"a61f0d11-698f-4f14-b46c-2a43e73e571b\" (UID: \"a61f0d11-698f-4f14-b46c-2a43e73e571b\") " Sep 15 12:05:21.606700 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.606540 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54zd5\" (UniqueName: \"kubernetes.io/projected/9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132-kube-api-access-54zd5\") pod \"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132\" (UID: \"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132\") " Sep 15 12:05:21.606920 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.606892 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61f0d11-698f-4f14-b46c-2a43e73e571b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a61f0d11-698f-4f14-b46c-2a43e73e571b" (UID: "a61f0d11-698f-4f14-b46c-2a43e73e571b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:05:21.606975 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.606891 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132" (UID: "9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:05:21.608639 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.608609 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132-kube-api-access-54zd5" (OuterVolumeSpecName: "kube-api-access-54zd5") pod "9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132" (UID: "9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132"). InnerVolumeSpecName "kube-api-access-54zd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:05:21.608754 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.608672 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61f0d11-698f-4f14-b46c-2a43e73e571b-kube-api-access-6nq8z" (OuterVolumeSpecName: "kube-api-access-6nq8z") pod "a61f0d11-698f-4f14-b46c-2a43e73e571b" (UID: "a61f0d11-698f-4f14-b46c-2a43e73e571b"). InnerVolumeSpecName "kube-api-access-6nq8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:05:21.707819 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.707772 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:21.707819 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.707813 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nq8z\" (UniqueName: \"kubernetes.io/projected/a61f0d11-698f-4f14-b46c-2a43e73e571b-kube-api-access-6nq8z\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:21.707819 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.707822 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61f0d11-698f-4f14-b46c-2a43e73e571b-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:21.707819 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:21.707832 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54zd5\" (UniqueName: \"kubernetes.io/projected/9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132-kube-api-access-54zd5\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:22.256918 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.256885 2573 generic.go:356] "Generic (PLEG): container finished" podID="9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132" containerID="1a65c18aefc82b479429d770bf35f391cd2ae86d45ca41dd4b39bb1e51157fb7" exitCode=0 Sep 15 12:05:22.257416 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.256929 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" Sep 15 12:05:22.257416 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.256972 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" event={"ID":"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132","Type":"ContainerDied","Data":"1a65c18aefc82b479429d770bf35f391cd2ae86d45ca41dd4b39bb1e51157fb7"} Sep 15 12:05:22.257416 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.257041 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd" event={"ID":"9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132","Type":"ContainerDied","Data":"34852df419415b1d1b4d23fe9914624a4fdd826ab62a1d8d087e9eb9122bdb2a"} Sep 15 12:05:22.257416 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.257067 2573 scope.go:117] "RemoveContainer" containerID="1a65c18aefc82b479429d770bf35f391cd2ae86d45ca41dd4b39bb1e51157fb7" Sep 15 12:05:22.258349 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.258327 2573 generic.go:356] "Generic (PLEG): container finished" podID="a61f0d11-698f-4f14-b46c-2a43e73e571b" containerID="94ed4e8f9e666bf09b797dd3492fc214d0388e215a29bebb1bcb27ea6386692d" exitCode=0 Sep 15 12:05:22.258472 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.258355 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" event={"ID":"a61f0d11-698f-4f14-b46c-2a43e73e571b","Type":"ContainerDied","Data":"94ed4e8f9e666bf09b797dd3492fc214d0388e215a29bebb1bcb27ea6386692d"} Sep 15 12:05:22.258472 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.258420 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" event={"ID":"a61f0d11-698f-4f14-b46c-2a43e73e571b","Type":"ContainerDied","Data":"deeadf4772746e32efd43736d98033712e0155218d7f2b84eebde25a230fd492"} Sep 15 12:05:22.258472 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.258372 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4" Sep 15 12:05:22.266588 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.266564 2573 scope.go:117] "RemoveContainer" containerID="1a65c18aefc82b479429d770bf35f391cd2ae86d45ca41dd4b39bb1e51157fb7" Sep 15 12:05:22.266898 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:05:22.266873 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a65c18aefc82b479429d770bf35f391cd2ae86d45ca41dd4b39bb1e51157fb7\": container with ID starting with 1a65c18aefc82b479429d770bf35f391cd2ae86d45ca41dd4b39bb1e51157fb7 not found: ID does not exist" containerID="1a65c18aefc82b479429d770bf35f391cd2ae86d45ca41dd4b39bb1e51157fb7" Sep 15 12:05:22.266982 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.266903 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a65c18aefc82b479429d770bf35f391cd2ae86d45ca41dd4b39bb1e51157fb7"} err="failed to get container status \"1a65c18aefc82b479429d770bf35f391cd2ae86d45ca41dd4b39bb1e51157fb7\": rpc error: code = NotFound desc = could not find container \"1a65c18aefc82b479429d770bf35f391cd2ae86d45ca41dd4b39bb1e51157fb7\": container with ID starting with 1a65c18aefc82b479429d770bf35f391cd2ae86d45ca41dd4b39bb1e51157fb7 not found: ID does not exist" Sep 15 12:05:22.266982 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.266921 2573 scope.go:117] "RemoveContainer" containerID="94ed4e8f9e666bf09b797dd3492fc214d0388e215a29bebb1bcb27ea6386692d" Sep 15 12:05:22.274777 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.274583 2573 scope.go:117] "RemoveContainer" containerID="94ed4e8f9e666bf09b797dd3492fc214d0388e215a29bebb1bcb27ea6386692d" Sep 15 12:05:22.274865 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:05:22.274845 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ed4e8f9e666bf09b797dd3492fc214d0388e215a29bebb1bcb27ea6386692d\": container with ID starting with 94ed4e8f9e666bf09b797dd3492fc214d0388e215a29bebb1bcb27ea6386692d not found: ID does not exist" containerID="94ed4e8f9e666bf09b797dd3492fc214d0388e215a29bebb1bcb27ea6386692d" Sep 15 12:05:22.274949 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.274872 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ed4e8f9e666bf09b797dd3492fc214d0388e215a29bebb1bcb27ea6386692d"} err="failed to get container status \"94ed4e8f9e666bf09b797dd3492fc214d0388e215a29bebb1bcb27ea6386692d\": rpc error: code = NotFound desc = could not find container \"94ed4e8f9e666bf09b797dd3492fc214d0388e215a29bebb1bcb27ea6386692d\": container with ID starting with 94ed4e8f9e666bf09b797dd3492fc214d0388e215a29bebb1bcb27ea6386692d not found: ID does not exist" Sep 15 12:05:22.279792 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.279773 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd"] Sep 15 12:05:22.286273 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.286252 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-42b26e-5bfbb77c9-bkzjd"] Sep 15 12:05:22.297853 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.297831 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4"] Sep 15 12:05:22.301590 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:22.301570 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-42b26e-5f9b444654-s4bw4"] Sep 15 12:05:23.234012 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:23.233975 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="501c647e-fdfb-45ad-a789-ffb878d4bbef" path="/var/lib/kubelet/pods/501c647e-fdfb-45ad-a789-ffb878d4bbef/volumes" Sep 15 12:05:23.234362 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:23.234346 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132" path="/var/lib/kubelet/pods/9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132/volumes" Sep 15 12:05:23.234663 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:23.234652 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a61f0d11-698f-4f14-b46c-2a43e73e571b" path="/var/lib/kubelet/pods/a61f0d11-698f-4f14-b46c-2a43e73e571b/volumes" Sep 15 12:05:23.234948 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:23.234936 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3f312de-7d08-438d-9650-48c90596e18a" path="/var/lib/kubelet/pods/c3f312de-7d08-438d-9650-48c90596e18a/volumes" Sep 15 12:05:28.589092 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.589057 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v"] Sep 15 12:05:28.589576 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.589549 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a61f0d11-698f-4f14-b46c-2a43e73e571b" containerName="payload-pre-processing" Sep 15 12:05:28.589576 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.589568 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61f0d11-698f-4f14-b46c-2a43e73e571b" containerName="payload-pre-processing" Sep 15 12:05:28.589697 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.589583 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132" containerName="payload-processing" Sep 15 12:05:28.589697 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.589592 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132" containerName="payload-processing" Sep 15 12:05:28.589697 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.589618 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f312de-7d08-438d-9650-48c90596e18a" containerName="revoke-keys" Sep 15 12:05:28.589697 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.589627 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f312de-7d08-438d-9650-48c90596e18a" containerName="revoke-keys" Sep 15 12:05:28.589896 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.589709 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f312de-7d08-438d-9650-48c90596e18a" containerName="revoke-keys" Sep 15 12:05:28.589896 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.589721 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a61f0d11-698f-4f14-b46c-2a43e73e571b" containerName="payload-pre-processing" Sep 15 12:05:28.589896 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.589735 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d1200c8-efbb-4caf-ae6f-a7fd3ee2d132" containerName="payload-processing" Sep 15 12:05:28.592615 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.592595 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v" Sep 15 12:05:28.595124 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.595095 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:05:28.595124 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.595100 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wjx8x\"" Sep 15 12:05:28.599900 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.599874 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v"] Sep 15 12:05:28.666389 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.666362 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291c8bdd-03d6-4ac0-95e0-953b87b9f588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v\" (UID: \"291c8bdd-03d6-4ac0-95e0-953b87b9f588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v" Sep 15 12:05:28.767320 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.767288 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291c8bdd-03d6-4ac0-95e0-953b87b9f588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v\" (UID: \"291c8bdd-03d6-4ac0-95e0-953b87b9f588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v" Sep 15 12:05:28.767869 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.767851 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291c8bdd-03d6-4ac0-95e0-953b87b9f588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v\" (UID: \"291c8bdd-03d6-4ac0-95e0-953b87b9f588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v" Sep 15 12:05:28.904040 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:28.903948 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v" Sep 15 12:05:29.172308 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:29.172226 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v"] Sep 15 12:05:29.175479 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:05:29.175448 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291c8bdd_03d6_4ac0_95e0_953b87b9f588.slice/crio-8271224260dcb500a4e04d3c573cc062d5cdb778476f6bca98bfc8e42ee7ecb1 WatchSource:0}: Error finding container 8271224260dcb500a4e04d3c573cc062d5cdb778476f6bca98bfc8e42ee7ecb1: Status 404 returned error can't find the container with id 8271224260dcb500a4e04d3c573cc062d5cdb778476f6bca98bfc8e42ee7ecb1 Sep 15 12:05:29.293620 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:29.293588 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v" event={"ID":"291c8bdd-03d6-4ac0-95e0-953b87b9f588","Type":"ContainerStarted","Data":"8271224260dcb500a4e04d3c573cc062d5cdb778476f6bca98bfc8e42ee7ecb1"} Sep 15 12:05:30.300578 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:30.300545 2573 generic.go:356] "Generic (PLEG): container finished" podID="291c8bdd-03d6-4ac0-95e0-953b87b9f588" containerID="dac3f3ad38aef219f1dc0041d7c9c1176f30691bcc91b764a10dc8f7f7d90f6b" exitCode=0 Sep 15 12:05:30.300953 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:30.300636 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v" event={"ID":"291c8bdd-03d6-4ac0-95e0-953b87b9f588","Type":"ContainerDied","Data":"dac3f3ad38aef219f1dc0041d7c9c1176f30691bcc91b764a10dc8f7f7d90f6b"} Sep 15 12:05:31.426545 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:31.426501 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v" Sep 15 12:05:31.492440 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:31.492395 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291c8bdd-03d6-4ac0-95e0-953b87b9f588-maas-api-service-ca\") pod \"291c8bdd-03d6-4ac0-95e0-953b87b9f588\" (UID: \"291c8bdd-03d6-4ac0-95e0-953b87b9f588\") " Sep 15 12:05:31.492771 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:31.492747 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291c8bdd-03d6-4ac0-95e0-953b87b9f588-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "291c8bdd-03d6-4ac0-95e0-953b87b9f588" (UID: "291c8bdd-03d6-4ac0-95e0-953b87b9f588"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:05:31.593489 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:31.593402 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291c8bdd-03d6-4ac0-95e0-953b87b9f588-maas-api-service-ca\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:32.311530 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:32.311494 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v" event={"ID":"291c8bdd-03d6-4ac0-95e0-953b87b9f588","Type":"ContainerDied","Data":"8271224260dcb500a4e04d3c573cc062d5cdb778476f6bca98bfc8e42ee7ecb1"} Sep 15 12:05:32.311707 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:32.311548 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8271224260dcb500a4e04d3c573cc062d5cdb778476f6bca98bfc8e42ee7ecb1" Sep 15 12:05:32.311707 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:32.311505 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v" Sep 15 12:05:39.387522 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:39.387481 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5"] Sep 15 12:05:39.387986 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:39.387716 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" podUID="99a48997-c5e1-4ac9-8664-465c0373e9f1" containerName="payload-pre-processing" containerID="cri-o://5657f869e396b7548d787c27ffad4882abeb6c8f5507b38b25bfb185f73269dc" gracePeriod=30 Sep 15 12:05:39.630538 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:39.630511 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v"] Sep 15 12:05:39.632575 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:39.632552 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" Sep 15 12:05:39.632789 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:39.632768 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b7b3c-cpk6v"] Sep 15 12:05:39.767742 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:39.767716 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6rh5\" (UniqueName: \"kubernetes.io/projected/99a48997-c5e1-4ac9-8664-465c0373e9f1-kube-api-access-c6rh5\") pod \"99a48997-c5e1-4ac9-8664-465c0373e9f1\" (UID: \"99a48997-c5e1-4ac9-8664-465c0373e9f1\") " Sep 15 12:05:39.767815 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:39.767766 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a48997-c5e1-4ac9-8664-465c0373e9f1-plugins-config-volume\") pod \"99a48997-c5e1-4ac9-8664-465c0373e9f1\" (UID: \"99a48997-c5e1-4ac9-8664-465c0373e9f1\") " Sep 15 12:05:39.768211 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:39.768182 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a48997-c5e1-4ac9-8664-465c0373e9f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99a48997-c5e1-4ac9-8664-465c0373e9f1" (UID: "99a48997-c5e1-4ac9-8664-465c0373e9f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:05:39.769811 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:39.769787 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a48997-c5e1-4ac9-8664-465c0373e9f1-kube-api-access-c6rh5" (OuterVolumeSpecName: "kube-api-access-c6rh5") pod "99a48997-c5e1-4ac9-8664-465c0373e9f1" (UID: "99a48997-c5e1-4ac9-8664-465c0373e9f1"). InnerVolumeSpecName "kube-api-access-c6rh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:05:39.868820 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:39.868790 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6rh5\" (UniqueName: \"kubernetes.io/projected/99a48997-c5e1-4ac9-8664-465c0373e9f1-kube-api-access-c6rh5\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:39.868820 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:39.868817 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a48997-c5e1-4ac9-8664-465c0373e9f1-plugins-config-volume\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:05:40.357790 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:40.357749 2573 generic.go:356] "Generic (PLEG): container finished" podID="99a48997-c5e1-4ac9-8664-465c0373e9f1" containerID="5657f869e396b7548d787c27ffad4882abeb6c8f5507b38b25bfb185f73269dc" exitCode=0 Sep 15 12:05:40.357992 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:40.357835 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" event={"ID":"99a48997-c5e1-4ac9-8664-465c0373e9f1","Type":"ContainerDied","Data":"5657f869e396b7548d787c27ffad4882abeb6c8f5507b38b25bfb185f73269dc"} Sep 15 12:05:40.357992 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:40.357860 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" Sep 15 12:05:40.357992 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:40.357902 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5" event={"ID":"99a48997-c5e1-4ac9-8664-465c0373e9f1","Type":"ContainerDied","Data":"bb5d9f06f9a668a1bfbbb63556766f89013ea3cec2c635facc9b77392774f0d3"} Sep 15 12:05:40.357992 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:40.357929 2573 scope.go:117] "RemoveContainer" containerID="5657f869e396b7548d787c27ffad4882abeb6c8f5507b38b25bfb185f73269dc" Sep 15 12:05:40.367384 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:40.367364 2573 scope.go:117] "RemoveContainer" containerID="5657f869e396b7548d787c27ffad4882abeb6c8f5507b38b25bfb185f73269dc" Sep 15 12:05:40.367707 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:05:40.367676 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5657f869e396b7548d787c27ffad4882abeb6c8f5507b38b25bfb185f73269dc\": container with ID starting with 5657f869e396b7548d787c27ffad4882abeb6c8f5507b38b25bfb185f73269dc not found: ID does not exist" containerID="5657f869e396b7548d787c27ffad4882abeb6c8f5507b38b25bfb185f73269dc" Sep 15 12:05:40.367839 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:40.367716 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5657f869e396b7548d787c27ffad4882abeb6c8f5507b38b25bfb185f73269dc"} err="failed to get container status \"5657f869e396b7548d787c27ffad4882abeb6c8f5507b38b25bfb185f73269dc\": rpc error: code = NotFound desc = could not find container \"5657f869e396b7548d787c27ffad4882abeb6c8f5507b38b25bfb185f73269dc\": container with ID starting with 5657f869e396b7548d787c27ffad4882abeb6c8f5507b38b25bfb185f73269dc not found: ID does not exist" Sep 15 12:05:40.385158 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:40.385131 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5"] Sep 15 12:05:40.392761 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:40.392738 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b7b3c-688574886-bbrr5"] Sep 15 12:05:41.234270 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:41.234241 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="291c8bdd-03d6-4ac0-95e0-953b87b9f588" path="/var/lib/kubelet/pods/291c8bdd-03d6-4ac0-95e0-953b87b9f588/volumes" Sep 15 12:05:41.234600 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:05:41.234587 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99a48997-c5e1-4ac9-8664-465c0373e9f1" path="/var/lib/kubelet/pods/99a48997-c5e1-4ac9-8664-465c0373e9f1/volumes" Sep 15 12:06:53.335639 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:06:53.335609 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 12:06:53.340403 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:06:53.340372 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 12:06:53.344162 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:06:53.344143 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 12:06:53.348720 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:06:53.348704 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 12:06:53.549229 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:06:53.549200 2573 scope.go:117] "RemoveContainer" containerID="ca7670e6cb465b118bc76de270b5e10dc9475dbca153ea6491163efed4a67bdb" Sep 15 12:06:53.557445 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:06:53.557427 2573 scope.go:117] "RemoveContainer" containerID="edf91999c9bd1da7e7ab633b5911bf9614ca9ac540adabe7793e5ff517d0a538" Sep 15 12:08:53.584736 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:08:53.584701 2573 scope.go:117] "RemoveContainer" containerID="538b1caed64a876881b7a60a49660c0e4d93ccc00b8c3ea35c5118f6aa8afa98" Sep 15 12:10:02.482304 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.482229 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d"] Sep 15 12:10:02.482759 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.482562 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99a48997-c5e1-4ac9-8664-465c0373e9f1" containerName="payload-pre-processing" Sep 15 12:10:02.482759 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.482573 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="99a48997-c5e1-4ac9-8664-465c0373e9f1" containerName="payload-pre-processing" Sep 15 12:10:02.482759 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.482583 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291c8bdd-03d6-4ac0-95e0-953b87b9f588" containerName="revoke-keys" Sep 15 12:10:02.482759 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.482588 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="291c8bdd-03d6-4ac0-95e0-953b87b9f588" containerName="revoke-keys" Sep 15 12:10:02.482759 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.482642 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="291c8bdd-03d6-4ac0-95e0-953b87b9f588" containerName="revoke-keys" Sep 15 12:10:02.482759 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.482652 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="99a48997-c5e1-4ac9-8664-465c0373e9f1" containerName="payload-pre-processing" Sep 15 12:10:02.485173 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.485155 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.488135 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.488113 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-29a099dc-kserve-self-signed-certs\"" Sep 15 12:10:02.496887 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.496862 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d"] Sep 15 12:10:02.516971 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.516947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37eceeab-db77-4b00-899d-ff40415d7e3a-tls-certs\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.517138 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.516996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-dshm\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.517138 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.517073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtqsw\" (UniqueName: \"kubernetes.io/projected/37eceeab-db77-4b00-899d-ff40415d7e3a-kube-api-access-jtqsw\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.517138 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.517114 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-home\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.517252 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.517151 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-model-cache\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.517252 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.517170 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-tmp-dir\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.617972 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.617931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-home\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.617972 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.617972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-model-cache\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.618217 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.617991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-tmp-dir\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.618217 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.618066 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37eceeab-db77-4b00-899d-ff40415d7e3a-tls-certs\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.618217 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.618129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-dshm\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.618217 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.618161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtqsw\" (UniqueName: \"kubernetes.io/projected/37eceeab-db77-4b00-899d-ff40415d7e3a-kube-api-access-jtqsw\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.618429 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.618336 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-home\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.618429 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.618411 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-model-cache\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.618539 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.618450 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-tmp-dir\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.620414 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.620394 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-dshm\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.620604 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.620585 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37eceeab-db77-4b00-899d-ff40415d7e3a-tls-certs\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.630344 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.630323 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtqsw\" (UniqueName: \"kubernetes.io/projected/37eceeab-db77-4b00-899d-ff40415d7e3a-kube-api-access-jtqsw\") pod \"e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:02.795065 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:02.795013 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:03.072255 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:03.072229 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d"] Sep 15 12:10:03.074791 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:10:03.074763 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37eceeab_db77_4b00_899d_ff40415d7e3a.slice/crio-2d0217c9b3942086a8e4c19caaca781f8f87aa9bf7e3e2b4f7b36fa2d19d21a3 WatchSource:0}: Error finding container 2d0217c9b3942086a8e4c19caaca781f8f87aa9bf7e3e2b4f7b36fa2d19d21a3: Status 404 returned error can't find the container with id 2d0217c9b3942086a8e4c19caaca781f8f87aa9bf7e3e2b4f7b36fa2d19d21a3 Sep 15 12:10:03.522418 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:03.522386 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" event={"ID":"37eceeab-db77-4b00-899d-ff40415d7e3a","Type":"ContainerStarted","Data":"a29f03b6edd8862a8bc25f7e0db13517175aa9aac166435a2663399686ca261d"} Sep 15 12:10:03.522814 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:03.522427 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" event={"ID":"37eceeab-db77-4b00-899d-ff40415d7e3a","Type":"ContainerStarted","Data":"2d0217c9b3942086a8e4c19caaca781f8f87aa9bf7e3e2b4f7b36fa2d19d21a3"} Sep 15 12:10:03.541152 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:03.541107 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" podStartSLOduration=1.541093297 podStartE2EDuration="1.541093297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:03.54021055 +0000 UTC m=+1990.770137085" watchObservedRunningTime="2026-09-15 12:10:03.541093297 +0000 UTC m=+1990.771019833" Sep 15 12:10:12.795819 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:12.795789 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:12.796544 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:12.795833 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:12.808668 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:12.808643 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:13.577277 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:13.577249 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:14.172092 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.172061 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c"] Sep 15 12:10:14.176378 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.176344 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.179508 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.179484 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-29a099dc-kserve-self-signed-certs\"" Sep 15 12:10:14.185246 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.185225 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c"] Sep 15 12:10:14.319929 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.319895 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk4db\" (UniqueName: \"kubernetes.io/projected/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-kube-api-access-kk4db\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.320117 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.319944 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-dshm\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.320117 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.319962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-tmp-dir\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.320117 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.320071 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-home\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.320117 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.320109 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-tls-certs\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.320267 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.320172 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-model-cache\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.421490 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.421461 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-model-cache\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.421649 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.421511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4db\" (UniqueName: \"kubernetes.io/projected/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-kube-api-access-kk4db\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.421649 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.421544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-dshm\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.421649 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.421559 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-tmp-dir\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.421649 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.421581 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-home\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.421649 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.421608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-tls-certs\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.421972 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.421941 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-model-cache\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.422075 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.422059 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-home\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.422273 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.422120 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-tmp-dir\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.423862 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.423832 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-dshm\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.424124 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.424108 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-tls-certs\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.429956 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.429934 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4db\" (UniqueName: \"kubernetes.io/projected/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-kube-api-access-kk4db\") pod \"e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.487806 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.487778 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:14.756588 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:14.756554 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c"] Sep 15 12:10:14.759844 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:10:14.759812 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ba831fa_62f6_4a92_89dc_f83c466d3d4b.slice/crio-8ad456b1e790301ac66b9ead551281adfc5488a40a3685b356aafe97236bcdba WatchSource:0}: Error finding container 8ad456b1e790301ac66b9ead551281adfc5488a40a3685b356aafe97236bcdba: Status 404 returned error can't find the container with id 8ad456b1e790301ac66b9ead551281adfc5488a40a3685b356aafe97236bcdba Sep 15 12:10:15.575823 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:15.575782 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" event={"ID":"6ba831fa-62f6-4a92-89dc-f83c466d3d4b","Type":"ContainerStarted","Data":"73f9f21b9ca7c2ff2b2ccf3a3f1c78e8ceef458979333ae705e8584518a4576a"} Sep 15 12:10:15.575823 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:15.575826 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" event={"ID":"6ba831fa-62f6-4a92-89dc-f83c466d3d4b","Type":"ContainerStarted","Data":"8ad456b1e790301ac66b9ead551281adfc5488a40a3685b356aafe97236bcdba"} Sep 15 12:10:15.594002 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:15.593958 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" podStartSLOduration=1.59394367 podStartE2EDuration="1.59394367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:15.592964579 +0000 UTC m=+2002.822891116" watchObservedRunningTime="2026-09-15 12:10:15.59394367 +0000 UTC m=+2002.823870205" Sep 15 12:10:24.488824 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:24.488788 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:24.489326 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:24.488840 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:24.501300 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:24.501277 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:24.627553 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:24.627526 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:34.902231 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:34.902196 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d"] Sep 15 12:10:35.995174 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:35.995128 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c"] Sep 15 12:10:49.921767 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:49.921724 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" podUID="37eceeab-db77-4b00-899d-ff40415d7e3a" containerName="main" containerID="cri-o://a29f03b6edd8862a8bc25f7e0db13517175aa9aac166435a2663399686ca261d" gracePeriod=45 Sep 15 12:10:50.167495 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.167464 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:50.229099 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.228995 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-home\") pod \"37eceeab-db77-4b00-899d-ff40415d7e3a\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " Sep 15 12:10:50.229099 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.229049 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-tmp-dir\") pod \"37eceeab-db77-4b00-899d-ff40415d7e3a\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " Sep 15 12:10:50.229325 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.229110 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37eceeab-db77-4b00-899d-ff40415d7e3a-tls-certs\") pod \"37eceeab-db77-4b00-899d-ff40415d7e3a\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " Sep 15 12:10:50.229325 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.229268 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-home" (OuterVolumeSpecName: "home") pod "37eceeab-db77-4b00-899d-ff40415d7e3a" (UID: "37eceeab-db77-4b00-899d-ff40415d7e3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:10:50.229325 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.229312 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtqsw\" (UniqueName: \"kubernetes.io/projected/37eceeab-db77-4b00-899d-ff40415d7e3a-kube-api-access-jtqsw\") pod \"37eceeab-db77-4b00-899d-ff40415d7e3a\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " Sep 15 12:10:50.229567 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.229356 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-dshm\") pod \"37eceeab-db77-4b00-899d-ff40415d7e3a\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " Sep 15 12:10:50.229567 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.229359 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37eceeab-db77-4b00-899d-ff40415d7e3a" (UID: "37eceeab-db77-4b00-899d-ff40415d7e3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:10:50.229567 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.229375 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-model-cache\") pod \"37eceeab-db77-4b00-899d-ff40415d7e3a\" (UID: \"37eceeab-db77-4b00-899d-ff40415d7e3a\") " Sep 15 12:10:50.229732 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.229635 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-model-cache" (OuterVolumeSpecName: "model-cache") pod "37eceeab-db77-4b00-899d-ff40415d7e3a" (UID: "37eceeab-db77-4b00-899d-ff40415d7e3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:10:50.229732 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.229643 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-home\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:10:50.229732 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.229668 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-tmp-dir\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:10:50.231355 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.231330 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-dshm" (OuterVolumeSpecName: "dshm") pod "37eceeab-db77-4b00-899d-ff40415d7e3a" (UID: "37eceeab-db77-4b00-899d-ff40415d7e3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:10:50.231466 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.231412 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37eceeab-db77-4b00-899d-ff40415d7e3a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37eceeab-db77-4b00-899d-ff40415d7e3a" (UID: "37eceeab-db77-4b00-899d-ff40415d7e3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:10:50.231466 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.231416 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37eceeab-db77-4b00-899d-ff40415d7e3a-kube-api-access-jtqsw" (OuterVolumeSpecName: "kube-api-access-jtqsw") pod "37eceeab-db77-4b00-899d-ff40415d7e3a" (UID: "37eceeab-db77-4b00-899d-ff40415d7e3a"). InnerVolumeSpecName "kube-api-access-jtqsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:10:50.330875 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.330839 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37eceeab-db77-4b00-899d-ff40415d7e3a-tls-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:10:50.330875 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.330868 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtqsw\" (UniqueName: \"kubernetes.io/projected/37eceeab-db77-4b00-899d-ff40415d7e3a-kube-api-access-jtqsw\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:10:50.330875 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.330880 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-dshm\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:10:50.331131 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.330892 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37eceeab-db77-4b00-899d-ff40415d7e3a-model-cache\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:10:50.737895 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.737863 2573 generic.go:356] "Generic (PLEG): container finished" podID="37eceeab-db77-4b00-899d-ff40415d7e3a" containerID="a29f03b6edd8862a8bc25f7e0db13517175aa9aac166435a2663399686ca261d" exitCode=0 Sep 15 12:10:50.738080 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.737905 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" Sep 15 12:10:50.738080 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.737950 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" event={"ID":"37eceeab-db77-4b00-899d-ff40415d7e3a","Type":"ContainerDied","Data":"a29f03b6edd8862a8bc25f7e0db13517175aa9aac166435a2663399686ca261d"} Sep 15 12:10:50.738080 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.737994 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d" event={"ID":"37eceeab-db77-4b00-899d-ff40415d7e3a","Type":"ContainerDied","Data":"2d0217c9b3942086a8e4c19caaca781f8f87aa9bf7e3e2b4f7b36fa2d19d21a3"} Sep 15 12:10:50.738080 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.738036 2573 scope.go:117] "RemoveContainer" containerID="a29f03b6edd8862a8bc25f7e0db13517175aa9aac166435a2663399686ca261d" Sep 15 12:10:50.746517 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.746502 2573 scope.go:117] "RemoveContainer" containerID="a29f03b6edd8862a8bc25f7e0db13517175aa9aac166435a2663399686ca261d" Sep 15 12:10:50.746783 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:10:50.746766 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29f03b6edd8862a8bc25f7e0db13517175aa9aac166435a2663399686ca261d\": container with ID starting with a29f03b6edd8862a8bc25f7e0db13517175aa9aac166435a2663399686ca261d not found: ID does not exist" containerID="a29f03b6edd8862a8bc25f7e0db13517175aa9aac166435a2663399686ca261d" Sep 15 12:10:50.746850 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.746789 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29f03b6edd8862a8bc25f7e0db13517175aa9aac166435a2663399686ca261d"} err="failed to get container status \"a29f03b6edd8862a8bc25f7e0db13517175aa9aac166435a2663399686ca261d\": rpc error: code = NotFound desc = could not find container \"a29f03b6edd8862a8bc25f7e0db13517175aa9aac166435a2663399686ca261d\": container with ID starting with a29f03b6edd8862a8bc25f7e0db13517175aa9aac166435a2663399686ca261d not found: ID does not exist" Sep 15 12:10:50.758694 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.758662 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d"] Sep 15 12:10:50.761774 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:50.761753 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-29a099dc-kserve-6568d7f98c-2k76d"] Sep 15 12:10:51.013769 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.013728 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" podUID="6ba831fa-62f6-4a92-89dc-f83c466d3d4b" containerName="main" containerID="cri-o://73f9f21b9ca7c2ff2b2ccf3a3f1c78e8ceef458979333ae705e8584518a4576a" gracePeriod=45 Sep 15 12:10:51.235290 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.235257 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37eceeab-db77-4b00-899d-ff40415d7e3a" path="/var/lib/kubelet/pods/37eceeab-db77-4b00-899d-ff40415d7e3a/volumes" Sep 15 12:10:51.259466 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.259446 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:51.338380 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.338302 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-home\") pod \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " Sep 15 12:10:51.338380 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.338350 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-model-cache\") pod \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " Sep 15 12:10:51.338599 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.338387 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-tmp-dir\") pod \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " Sep 15 12:10:51.338599 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.338427 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-tls-certs\") pod \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " Sep 15 12:10:51.338599 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.338452 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-dshm\") pod \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " Sep 15 12:10:51.338599 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.338519 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk4db\" (UniqueName: \"kubernetes.io/projected/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-kube-api-access-kk4db\") pod \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\" (UID: \"6ba831fa-62f6-4a92-89dc-f83c466d3d4b\") " Sep 15 12:10:51.338599 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.338556 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-home" (OuterVolumeSpecName: "home") pod "6ba831fa-62f6-4a92-89dc-f83c466d3d4b" (UID: "6ba831fa-62f6-4a92-89dc-f83c466d3d4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:10:51.338825 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.338671 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ba831fa-62f6-4a92-89dc-f83c466d3d4b" (UID: "6ba831fa-62f6-4a92-89dc-f83c466d3d4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:10:51.338825 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.338699 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ba831fa-62f6-4a92-89dc-f83c466d3d4b" (UID: "6ba831fa-62f6-4a92-89dc-f83c466d3d4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:10:51.338825 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.338802 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-home\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:10:51.338825 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.338819 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-model-cache\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:10:51.338955 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.338835 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-tmp-dir\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:10:51.340578 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.340554 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ba831fa-62f6-4a92-89dc-f83c466d3d4b" (UID: "6ba831fa-62f6-4a92-89dc-f83c466d3d4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:10:51.340578 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.340564 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-kube-api-access-kk4db" (OuterVolumeSpecName: "kube-api-access-kk4db") pod "6ba831fa-62f6-4a92-89dc-f83c466d3d4b" (UID: "6ba831fa-62f6-4a92-89dc-f83c466d3d4b"). InnerVolumeSpecName "kube-api-access-kk4db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:10:51.340754 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.340562 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-dshm" (OuterVolumeSpecName: "dshm") pod "6ba831fa-62f6-4a92-89dc-f83c466d3d4b" (UID: "6ba831fa-62f6-4a92-89dc-f83c466d3d4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:10:51.440289 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.440254 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-tls-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:10:51.440289 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.440286 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-dshm\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:10:51.440466 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.440299 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk4db\" (UniqueName: \"kubernetes.io/projected/6ba831fa-62f6-4a92-89dc-f83c466d3d4b-kube-api-access-kk4db\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:10:51.744583 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.744497 2573 generic.go:356] "Generic (PLEG): container finished" podID="6ba831fa-62f6-4a92-89dc-f83c466d3d4b" containerID="73f9f21b9ca7c2ff2b2ccf3a3f1c78e8ceef458979333ae705e8584518a4576a" exitCode=0 Sep 15 12:10:51.744583 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.744533 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" event={"ID":"6ba831fa-62f6-4a92-89dc-f83c466d3d4b","Type":"ContainerDied","Data":"73f9f21b9ca7c2ff2b2ccf3a3f1c78e8ceef458979333ae705e8584518a4576a"} Sep 15 12:10:51.744583 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.744538 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" Sep 15 12:10:51.744583 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.744561 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c" event={"ID":"6ba831fa-62f6-4a92-89dc-f83c466d3d4b","Type":"ContainerDied","Data":"8ad456b1e790301ac66b9ead551281adfc5488a40a3685b356aafe97236bcdba"} Sep 15 12:10:51.744583 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.744578 2573 scope.go:117] "RemoveContainer" containerID="73f9f21b9ca7c2ff2b2ccf3a3f1c78e8ceef458979333ae705e8584518a4576a" Sep 15 12:10:51.753474 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.753457 2573 scope.go:117] "RemoveContainer" containerID="73f9f21b9ca7c2ff2b2ccf3a3f1c78e8ceef458979333ae705e8584518a4576a" Sep 15 12:10:51.753734 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:10:51.753714 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f9f21b9ca7c2ff2b2ccf3a3f1c78e8ceef458979333ae705e8584518a4576a\": container with ID starting with 73f9f21b9ca7c2ff2b2ccf3a3f1c78e8ceef458979333ae705e8584518a4576a not found: ID does not exist" containerID="73f9f21b9ca7c2ff2b2ccf3a3f1c78e8ceef458979333ae705e8584518a4576a" Sep 15 12:10:51.753799 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.753741 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f9f21b9ca7c2ff2b2ccf3a3f1c78e8ceef458979333ae705e8584518a4576a"} err="failed to get container status \"73f9f21b9ca7c2ff2b2ccf3a3f1c78e8ceef458979333ae705e8584518a4576a\": rpc error: code = NotFound desc = could not find container \"73f9f21b9ca7c2ff2b2ccf3a3f1c78e8ceef458979333ae705e8584518a4576a\": container with ID starting with 73f9f21b9ca7c2ff2b2ccf3a3f1c78e8ceef458979333ae705e8584518a4576a not found: ID does not exist" Sep 15 12:10:51.765430 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.765403 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c"] Sep 15 12:10:51.771334 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:51.771312 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-29a099dc-kserve-55f7688d4c-xm78c"] Sep 15 12:10:53.090174 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.090136 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w"] Sep 15 12:10:53.090563 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.090446 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba831fa-62f6-4a92-89dc-f83c466d3d4b" containerName="main" Sep 15 12:10:53.090563 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.090459 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba831fa-62f6-4a92-89dc-f83c466d3d4b" containerName="main" Sep 15 12:10:53.090563 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.090484 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37eceeab-db77-4b00-899d-ff40415d7e3a" containerName="main" Sep 15 12:10:53.090563 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.090490 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37eceeab-db77-4b00-899d-ff40415d7e3a" containerName="main" Sep 15 12:10:53.090563 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.090538 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba831fa-62f6-4a92-89dc-f83c466d3d4b" containerName="main" Sep 15 12:10:53.090563 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.090546 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="37eceeab-db77-4b00-899d-ff40415d7e3a" containerName="main" Sep 15 12:10:53.094916 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.094899 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.098633 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.098602 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9dhbv\"" Sep 15 12:10:53.098786 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.098656 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:10:53.098786 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.098683 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 15 12:10:53.098786 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.098656 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 15 12:10:53.098786 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.098732 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 15 12:10:53.104369 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.104337 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w"] Sep 15 12:10:53.153724 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.153690 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d7b96c-f03f-4677-822f-17ed048fb071-maas-api-tls\") pod \"maas-api-59f6566bbc-vpc9w\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.153887 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.153749 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d7b96c-f03f-4677-822f-17ed048fb071-openshift-service-ca\") pod \"maas-api-59f6566bbc-vpc9w\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.153887 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.153840 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07d7b96c-f03f-4677-822f-17ed048fb071-metrics-certs\") pod \"maas-api-59f6566bbc-vpc9w\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.153887 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.153866 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svhxp\" (UniqueName: \"kubernetes.io/projected/07d7b96c-f03f-4677-822f-17ed048fb071-kube-api-access-svhxp\") pod \"maas-api-59f6566bbc-vpc9w\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.235009 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.234973 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ba831fa-62f6-4a92-89dc-f83c466d3d4b" path="/var/lib/kubelet/pods/6ba831fa-62f6-4a92-89dc-f83c466d3d4b/volumes" Sep 15 12:10:53.254941 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.254910 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d7b96c-f03f-4677-822f-17ed048fb071-maas-api-tls\") pod \"maas-api-59f6566bbc-vpc9w\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.255123 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.254982 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d7b96c-f03f-4677-822f-17ed048fb071-openshift-service-ca\") pod \"maas-api-59f6566bbc-vpc9w\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.255123 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.255061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07d7b96c-f03f-4677-822f-17ed048fb071-metrics-certs\") pod \"maas-api-59f6566bbc-vpc9w\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.255123 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.255092 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhxp\" (UniqueName: \"kubernetes.io/projected/07d7b96c-f03f-4677-822f-17ed048fb071-kube-api-access-svhxp\") pod \"maas-api-59f6566bbc-vpc9w\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.257402 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.257376 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 15 12:10:53.257402 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.257396 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 15 12:10:53.257542 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.257423 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:10:53.262401 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.262383 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 15 12:10:53.266624 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.266605 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d7b96c-f03f-4677-822f-17ed048fb071-openshift-service-ca\") pod \"maas-api-59f6566bbc-vpc9w\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.268448 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.268421 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d7b96c-f03f-4677-822f-17ed048fb071-maas-api-tls\") pod \"maas-api-59f6566bbc-vpc9w\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.268530 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.268475 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07d7b96c-f03f-4677-822f-17ed048fb071-metrics-certs\") pod \"maas-api-59f6566bbc-vpc9w\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.273210 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.273189 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svhxp\" (UniqueName: \"kubernetes.io/projected/07d7b96c-f03f-4677-822f-17ed048fb071-kube-api-access-svhxp\") pod \"maas-api-59f6566bbc-vpc9w\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.409854 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.409775 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9dhbv\"" Sep 15 12:10:53.417132 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.417108 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:53.693627 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.693558 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w"] Sep 15 12:10:53.696929 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:10:53.696900 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d7b96c_f03f_4677_822f_17ed048fb071.slice/crio-1ce12bbe790b2c5d78902b748d2efc72f9b536050823f0a200240c953a141510 WatchSource:0}: Error finding container 1ce12bbe790b2c5d78902b748d2efc72f9b536050823f0a200240c953a141510: Status 404 returned error can't find the container with id 1ce12bbe790b2c5d78902b748d2efc72f9b536050823f0a200240c953a141510 Sep 15 12:10:53.698620 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.698601 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 12:10:53.755773 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:53.755734 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" event={"ID":"07d7b96c-f03f-4677-822f-17ed048fb071","Type":"ContainerStarted","Data":"1ce12bbe790b2c5d78902b748d2efc72f9b536050823f0a200240c953a141510"} Sep 15 12:10:55.767109 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:55.767061 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" event={"ID":"07d7b96c-f03f-4677-822f-17ed048fb071","Type":"ContainerStarted","Data":"f968b81e6a08652bfcaecadbcdafb976ba8a0c37019e1871cccd91cd314f8cec"} Sep 15 12:10:55.767604 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:55.767166 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:10:55.785641 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:10:55.785586 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" podStartSLOduration=1.540033372 podStartE2EDuration="2.785571237s" totalImagesPullingTime="1.245537865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:55.78334745 +0000 UTC m=+2043.013273987" watchObservedRunningTime="2026-09-15 12:10:55.785571237 +0000 UTC m=+2043.015497773" Sep 15 12:11:00.496153 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.496113 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w"] Sep 15 12:11:00.496561 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.496326 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" podUID="07d7b96c-f03f-4677-822f-17ed048fb071" containerName="maas-api" containerID="cri-o://f968b81e6a08652bfcaecadbcdafb976ba8a0c37019e1871cccd91cd314f8cec" gracePeriod=30 Sep 15 12:11:00.502111 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.502087 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:11:00.533423 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.533389 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz"] Sep 15 12:11:00.536714 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.536694 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.549371 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.549343 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz"] Sep 15 12:11:00.622380 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.622349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8679bc04-033d-4d26-b02c-67a6603d542e-maas-api-tls\") pod \"maas-api-86d9ddc875-lzmtz\" (UID: \"8679bc04-033d-4d26-b02c-67a6603d542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.622525 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.622402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n8wv\" (UniqueName: \"kubernetes.io/projected/8679bc04-033d-4d26-b02c-67a6603d542e-kube-api-access-2n8wv\") pod \"maas-api-86d9ddc875-lzmtz\" (UID: \"8679bc04-033d-4d26-b02c-67a6603d542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.622525 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.622452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8679bc04-033d-4d26-b02c-67a6603d542e-openshift-service-ca\") pod \"maas-api-86d9ddc875-lzmtz\" (UID: \"8679bc04-033d-4d26-b02c-67a6603d542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.622525 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.622507 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8679bc04-033d-4d26-b02c-67a6603d542e-metrics-certs\") pod \"maas-api-86d9ddc875-lzmtz\" (UID: \"8679bc04-033d-4d26-b02c-67a6603d542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.723403 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.723372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8679bc04-033d-4d26-b02c-67a6603d542e-metrics-certs\") pod \"maas-api-86d9ddc875-lzmtz\" (UID: \"8679bc04-033d-4d26-b02c-67a6603d542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.723571 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.723438 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8679bc04-033d-4d26-b02c-67a6603d542e-maas-api-tls\") pod \"maas-api-86d9ddc875-lzmtz\" (UID: \"8679bc04-033d-4d26-b02c-67a6603d542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.723571 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.723481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n8wv\" (UniqueName: \"kubernetes.io/projected/8679bc04-033d-4d26-b02c-67a6603d542e-kube-api-access-2n8wv\") pod \"maas-api-86d9ddc875-lzmtz\" (UID: \"8679bc04-033d-4d26-b02c-67a6603d542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.723571 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.723507 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8679bc04-033d-4d26-b02c-67a6603d542e-openshift-service-ca\") pod \"maas-api-86d9ddc875-lzmtz\" (UID: \"8679bc04-033d-4d26-b02c-67a6603d542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.724177 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.724157 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8679bc04-033d-4d26-b02c-67a6603d542e-openshift-service-ca\") pod \"maas-api-86d9ddc875-lzmtz\" (UID: \"8679bc04-033d-4d26-b02c-67a6603d542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.725897 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.725873 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8679bc04-033d-4d26-b02c-67a6603d542e-metrics-certs\") pod \"maas-api-86d9ddc875-lzmtz\" (UID: \"8679bc04-033d-4d26-b02c-67a6603d542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.726001 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.725960 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8679bc04-033d-4d26-b02c-67a6603d542e-maas-api-tls\") pod \"maas-api-86d9ddc875-lzmtz\" (UID: \"8679bc04-033d-4d26-b02c-67a6603d542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.733863 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.733840 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n8wv\" (UniqueName: \"kubernetes.io/projected/8679bc04-033d-4d26-b02c-67a6603d542e-kube-api-access-2n8wv\") pod \"maas-api-86d9ddc875-lzmtz\" (UID: \"8679bc04-033d-4d26-b02c-67a6603d542e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.748477 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.748460 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:11:00.789788 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.789749 2573 generic.go:356] "Generic (PLEG): container finished" podID="07d7b96c-f03f-4677-822f-17ed048fb071" containerID="f968b81e6a08652bfcaecadbcdafb976ba8a0c37019e1871cccd91cd314f8cec" exitCode=0 Sep 15 12:11:00.789941 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.789792 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" Sep 15 12:11:00.789941 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.789830 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" event={"ID":"07d7b96c-f03f-4677-822f-17ed048fb071","Type":"ContainerDied","Data":"f968b81e6a08652bfcaecadbcdafb976ba8a0c37019e1871cccd91cd314f8cec"} Sep 15 12:11:00.789941 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.789887 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w" event={"ID":"07d7b96c-f03f-4677-822f-17ed048fb071","Type":"ContainerDied","Data":"1ce12bbe790b2c5d78902b748d2efc72f9b536050823f0a200240c953a141510"} Sep 15 12:11:00.789941 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.789911 2573 scope.go:117] "RemoveContainer" containerID="f968b81e6a08652bfcaecadbcdafb976ba8a0c37019e1871cccd91cd314f8cec" Sep 15 12:11:00.798639 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.798620 2573 scope.go:117] "RemoveContainer" containerID="f968b81e6a08652bfcaecadbcdafb976ba8a0c37019e1871cccd91cd314f8cec" Sep 15 12:11:00.798926 ip-10-0-128-189 kubenswrapper[2573]: E0915 12:11:00.798906 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f968b81e6a08652bfcaecadbcdafb976ba8a0c37019e1871cccd91cd314f8cec\": container with ID starting with f968b81e6a08652bfcaecadbcdafb976ba8a0c37019e1871cccd91cd314f8cec not found: ID does not exist" containerID="f968b81e6a08652bfcaecadbcdafb976ba8a0c37019e1871cccd91cd314f8cec" Sep 15 12:11:00.799002 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.798934 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f968b81e6a08652bfcaecadbcdafb976ba8a0c37019e1871cccd91cd314f8cec"} err="failed to get container status \"f968b81e6a08652bfcaecadbcdafb976ba8a0c37019e1871cccd91cd314f8cec\": rpc error: code = NotFound desc = could not find container \"f968b81e6a08652bfcaecadbcdafb976ba8a0c37019e1871cccd91cd314f8cec\": container with ID starting with f968b81e6a08652bfcaecadbcdafb976ba8a0c37019e1871cccd91cd314f8cec not found: ID does not exist" Sep 15 12:11:00.824298 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.824273 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07d7b96c-f03f-4677-822f-17ed048fb071-metrics-certs\") pod \"07d7b96c-f03f-4677-822f-17ed048fb071\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " Sep 15 12:11:00.824400 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.824312 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d7b96c-f03f-4677-822f-17ed048fb071-maas-api-tls\") pod \"07d7b96c-f03f-4677-822f-17ed048fb071\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " Sep 15 12:11:00.824400 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.824331 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d7b96c-f03f-4677-822f-17ed048fb071-openshift-service-ca\") pod \"07d7b96c-f03f-4677-822f-17ed048fb071\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " Sep 15 12:11:00.824400 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.824360 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svhxp\" (UniqueName: \"kubernetes.io/projected/07d7b96c-f03f-4677-822f-17ed048fb071-kube-api-access-svhxp\") pod \"07d7b96c-f03f-4677-822f-17ed048fb071\" (UID: \"07d7b96c-f03f-4677-822f-17ed048fb071\") " Sep 15 12:11:00.824691 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.824669 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d7b96c-f03f-4677-822f-17ed048fb071-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07d7b96c-f03f-4677-822f-17ed048fb071" (UID: "07d7b96c-f03f-4677-822f-17ed048fb071"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:11:00.826411 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.826386 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d7b96c-f03f-4677-822f-17ed048fb071-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "07d7b96c-f03f-4677-822f-17ed048fb071" (UID: "07d7b96c-f03f-4677-822f-17ed048fb071"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:11:00.826411 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.826403 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d7b96c-f03f-4677-822f-17ed048fb071-kube-api-access-svhxp" (OuterVolumeSpecName: "kube-api-access-svhxp") pod "07d7b96c-f03f-4677-822f-17ed048fb071" (UID: "07d7b96c-f03f-4677-822f-17ed048fb071"). InnerVolumeSpecName "kube-api-access-svhxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:11:00.826587 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.826403 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d7b96c-f03f-4677-822f-17ed048fb071-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07d7b96c-f03f-4677-822f-17ed048fb071" (UID: "07d7b96c-f03f-4677-822f-17ed048fb071"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:11:00.897971 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.897942 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:00.925387 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.925360 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07d7b96c-f03f-4677-822f-17ed048fb071-metrics-certs\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:11:00.925387 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.925385 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d7b96c-f03f-4677-822f-17ed048fb071-maas-api-tls\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:11:00.925541 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.925395 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d7b96c-f03f-4677-822f-17ed048fb071-openshift-service-ca\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:11:00.925541 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:00.925406 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svhxp\" (UniqueName: \"kubernetes.io/projected/07d7b96c-f03f-4677-822f-17ed048fb071-kube-api-access-svhxp\") on node \"ip-10-0-128-189.ec2.internal\" DevicePath \"\"" Sep 15 12:11:01.111314 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:01.111276 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w"] Sep 15 12:11:01.113456 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:01.113432 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-vpc9w"] Sep 15 12:11:01.168635 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:01.168611 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz"] Sep 15 12:11:01.171057 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:11:01.171015 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8679bc04_033d_4d26_b02c_67a6603d542e.slice/crio-b664ce952449d75170f2057cbc445e455a886f65385cd709b20f8ca97863688f WatchSource:0}: Error finding container b664ce952449d75170f2057cbc445e455a886f65385cd709b20f8ca97863688f: Status 404 returned error can't find the container with id b664ce952449d75170f2057cbc445e455a886f65385cd709b20f8ca97863688f Sep 15 12:11:01.234516 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:01.234483 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07d7b96c-f03f-4677-822f-17ed048fb071" path="/var/lib/kubelet/pods/07d7b96c-f03f-4677-822f-17ed048fb071/volumes" Sep 15 12:11:01.797344 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:01.797254 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" event={"ID":"8679bc04-033d-4d26-b02c-67a6603d542e","Type":"ContainerStarted","Data":"9335ec1da43e7cc0f3a4d79115cedfcdb7268deb2079283d3b0a1c58748678e4"} Sep 15 12:11:01.797344 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:01.797314 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" event={"ID":"8679bc04-033d-4d26-b02c-67a6603d542e","Type":"ContainerStarted","Data":"b664ce952449d75170f2057cbc445e455a886f65385cd709b20f8ca97863688f"} Sep 15 12:11:01.797741 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:01.797353 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:01.820649 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:01.820598 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" podStartSLOduration=1.450160478 podStartE2EDuration="1.820582014s" totalImagesPullingTime="370.421536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:11:01.819431422 +0000 UTC m=+2049.049357957" watchObservedRunningTime="2026-09-15 12:11:01.820582014 +0000 UTC m=+2049.050508606" Sep 15 12:11:07.806721 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:07.806677 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-lzmtz" Sep 15 12:11:25.507940 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:25.507843 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-msl5t_ecdd2ee0-6fba-4398-ad1b-87c1a4dca820/global-pull-secret-syncer/0.log" Sep 15 12:11:25.647177 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:25.647144 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xqnpw_7787af06-5984-4095-ac98-a920406f8b40/konnectivity-agent/0.log" Sep 15 12:11:25.669841 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:25.669819 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-189.ec2.internal_0cce461e99c8c476d4a7a451016bffc0/haproxy/0.log" Sep 15 12:11:30.619725 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:30.619701 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9vw8p_51b2effb-31d6-4b96-b9a0-c9801353cb00/kuadrant-console-plugin/0.log" Sep 15 12:11:32.599655 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:32.599575 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-n6tc5_bbdaac6c-adf2-43e0-ae77-954292189938/cluster-monitoring-operator/0.log" Sep 15 12:11:32.856512 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:32.856420 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pngt7_e6d1bd46-92b5-4844-9d77-932851964761/node-exporter/0.log" Sep 15 12:11:32.880111 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:32.880068 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pngt7_e6d1bd46-92b5-4844-9d77-932851964761/kube-rbac-proxy/0.log" Sep 15 12:11:32.905322 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:32.905301 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pngt7_e6d1bd46-92b5-4844-9d77-932851964761/init-textfile/0.log" Sep 15 12:11:34.941600 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:34.941546 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-q58lq_dee2b36b-0dc3-4848-a413-ffca15516920/networking-console-plugin/0.log" Sep 15 12:11:35.536837 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:35.536789 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 12:11:35.541207 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:35.541186 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/2.log" Sep 15 12:11:36.806414 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:36.806383 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-2lz98_278b23ea-e4b2-440d-8c62-2cea30acb706/volume-data-source-validator/0.log" Sep 15 12:11:37.087814 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.087729 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l"] Sep 15 12:11:37.088089 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.088075 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d7b96c-f03f-4677-822f-17ed048fb071" containerName="maas-api" Sep 15 12:11:37.088145 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.088090 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d7b96c-f03f-4677-822f-17ed048fb071" containerName="maas-api" Sep 15 12:11:37.088182 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.088177 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d7b96c-f03f-4677-822f-17ed048fb071" containerName="maas-api" Sep 15 12:11:37.091213 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.091192 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.093534 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.093514 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5wxhw\"/\"openshift-service-ca.crt\"" Sep 15 12:11:37.094457 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.094431 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5wxhw\"/\"default-dockercfg-lkh2z\"" Sep 15 12:11:37.094569 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.094520 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5wxhw\"/\"kube-root-ca.crt\"" Sep 15 12:11:37.101160 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.101139 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l"] Sep 15 12:11:37.125987 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.125962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfqzt\" (UniqueName: \"kubernetes.io/projected/01958899-00dc-4ad4-88f5-0dbc8d702389-kube-api-access-mfqzt\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.126141 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.126080 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01958899-00dc-4ad4-88f5-0dbc8d702389-sys\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.126141 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.126135 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/01958899-00dc-4ad4-88f5-0dbc8d702389-proc\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.126259 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.126241 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01958899-00dc-4ad4-88f5-0dbc8d702389-lib-modules\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.126319 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.126278 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/01958899-00dc-4ad4-88f5-0dbc8d702389-podres\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.227386 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.227363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01958899-00dc-4ad4-88f5-0dbc8d702389-lib-modules\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.227524 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.227394 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/01958899-00dc-4ad4-88f5-0dbc8d702389-podres\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.227524 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.227418 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfqzt\" (UniqueName: \"kubernetes.io/projected/01958899-00dc-4ad4-88f5-0dbc8d702389-kube-api-access-mfqzt\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.227524 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.227449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01958899-00dc-4ad4-88f5-0dbc8d702389-sys\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.227524 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.227480 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/01958899-00dc-4ad4-88f5-0dbc8d702389-proc\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.227703 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.227553 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01958899-00dc-4ad4-88f5-0dbc8d702389-lib-modules\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.227703 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.227564 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/01958899-00dc-4ad4-88f5-0dbc8d702389-proc\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.227703 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.227574 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01958899-00dc-4ad4-88f5-0dbc8d702389-sys\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.227703 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.227554 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/01958899-00dc-4ad4-88f5-0dbc8d702389-podres\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.236248 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.236229 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfqzt\" (UniqueName: \"kubernetes.io/projected/01958899-00dc-4ad4-88f5-0dbc8d702389-kube-api-access-mfqzt\") pod \"perf-node-gather-daemonset-v4p4l\" (UID: \"01958899-00dc-4ad4-88f5-0dbc8d702389\") " pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.401848 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.401778 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.670153 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.670069 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l"] Sep 15 12:11:37.672940 ip-10-0-128-189 kubenswrapper[2573]: W0915 12:11:37.672908 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod01958899_00dc_4ad4_88f5_0dbc8d702389.slice/crio-879466f539c562610c91fdc9401d66d57a0ff7534973f98f29499f58f14b13ca WatchSource:0}: Error finding container 879466f539c562610c91fdc9401d66d57a0ff7534973f98f29499f58f14b13ca: Status 404 returned error can't find the container with id 879466f539c562610c91fdc9401d66d57a0ff7534973f98f29499f58f14b13ca Sep 15 12:11:37.961426 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.961338 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" event={"ID":"01958899-00dc-4ad4-88f5-0dbc8d702389","Type":"ContainerStarted","Data":"11b22f493b96b3283960742dd0aad9e3337731940390042486b8ad6cb96a7a5a"} Sep 15 12:11:37.961426 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.961391 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" event={"ID":"01958899-00dc-4ad4-88f5-0dbc8d702389","Type":"ContainerStarted","Data":"879466f539c562610c91fdc9401d66d57a0ff7534973f98f29499f58f14b13ca"} Sep 15 12:11:37.961820 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.961455 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:37.978798 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:37.978751 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" podStartSLOduration=0.978738131 podStartE2EDuration="978.738131ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:11:37.977214617 +0000 UTC m=+2085.207141158" watchObservedRunningTime="2026-09-15 12:11:37.978738131 +0000 UTC m=+2085.208664667" Sep 15 12:11:38.148472 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:38.148446 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94pfg_39b7b0de-0d7f-4438-bd33-6a189e691139/dns/0.log" Sep 15 12:11:38.168783 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:38.168761 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94pfg_39b7b0de-0d7f-4438-bd33-6a189e691139/kube-rbac-proxy/0.log" Sep 15 12:11:38.235619 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:38.235560 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z8grp_b4cec296-04c2-48ae-8024-09e47e456b59/dns-node-resolver/0.log" Sep 15 12:11:38.804427 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:38.804379 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2sll8_14fd92d1-ab33-45f3-b40f-10048dda0c33/node-ca/0.log" Sep 15 12:11:40.316508 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:40.316465 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-687d9c5d49-qph89_f6afee05-aacb-4d71-b6f2-57122da1548a/kube-auth-proxy/0.log" Sep 15 12:11:40.456225 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:40.456176 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5ddcf9f498-ghttm_78d07596-2598-4c06-a63c-ecb48c2052a0/router/0.log" Sep 15 12:11:41.329991 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:41.329961 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hp6hw_a645fa8f-dad2-486b-b3af-7ee707deaa93/serve-healthcheck-canary/0.log" Sep 15 12:11:41.854315 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:41.854284 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-hd9fz_22eeb9ed-5ded-442e-82ae-ac31f979e681/insights-operator/0.log" Sep 15 12:11:41.854504 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:41.854360 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-hd9fz_22eeb9ed-5ded-442e-82ae-ac31f979e681/insights-operator/1.log" Sep 15 12:11:41.875079 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:41.875056 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-599wc_3351650b-9731-4e0b-a025-e65b6a1633ff/kube-rbac-proxy/0.log" Sep 15 12:11:41.894678 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:41.894659 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-599wc_3351650b-9731-4e0b-a025-e65b6a1633ff/exporter/0.log" Sep 15 12:11:41.915612 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:41.915595 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-599wc_3351650b-9731-4e0b-a025-e65b6a1633ff/extractor/0.log" Sep 15 12:11:43.974823 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:43.974798 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5wxhw/perf-node-gather-daemonset-v4p4l" Sep 15 12:11:44.825732 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:44.825701 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74bdb8b799-8nzlk_47e85153-582d-4d5b-8c7e-1f56b2f6d0c8/manager/0.log" Sep 15 12:11:44.845487 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:44.845461 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74bdb8b799-8nzlk_47e85153-582d-4d5b-8c7e-1f56b2f6d0c8/copy-manifests/0.log" Sep 15 12:11:46.335805 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:46.335738 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8487688d7-g6mdd_2417650e-463e-429f-be11-127663b61d01/manager/0.log" Sep 15 12:11:46.663066 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:46.662970 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-687d9c5d49-qph89_f6afee05-aacb-4d71-b6f2-57122da1548a/kube-auth-proxy/0.log" Sep 15 12:11:46.770413 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:46.770382 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5ddcf9f498-ghttm_78d07596-2598-4c06-a63c-ecb48c2052a0/router/0.log" Sep 15 12:11:48.882510 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:48.882484 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-687d9c5d49-qph89_f6afee05-aacb-4d71-b6f2-57122da1548a/kube-auth-proxy/0.log" Sep 15 12:11:49.028188 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:49.028156 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5ddcf9f498-ghttm_78d07596-2598-4c06-a63c-ecb48c2052a0/router/0.log" Sep 15 12:11:52.102876 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:52.102835 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-6f89j_49259c18-d5c4-4a14-b715-60dae1b8a2ef/kube-storage-version-migrator-operator/1.log" Sep 15 12:11:52.103602 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:52.103582 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-6f89j_49259c18-d5c4-4a14-b715-60dae1b8a2ef/kube-storage-version-migrator-operator/0.log" Sep 15 12:11:53.313201 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.313172 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4mzd_4b6f665e-7431-4599-b9ef-302eb545b1c9/kube-multus-additional-cni-plugins/0.log" Sep 15 12:11:53.336848 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.336814 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4mzd_4b6f665e-7431-4599-b9ef-302eb545b1c9/egress-router-binary-copy/0.log" Sep 15 12:11:53.359454 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.359426 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4mzd_4b6f665e-7431-4599-b9ef-302eb545b1c9/cni-plugins/0.log" Sep 15 12:11:53.365338 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.365312 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 12:11:53.371297 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.371279 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nzs6t_590c2c95-948f-4758-8c73-c327a005adf1/console-operator/1.log" Sep 15 12:11:53.374396 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.374377 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 12:11:53.379504 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.379484 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 12:11:53.383061 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.383042 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4mzd_4b6f665e-7431-4599-b9ef-302eb545b1c9/bond-cni-plugin/0.log" Sep 15 12:11:53.403478 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.403457 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4mzd_4b6f665e-7431-4599-b9ef-302eb545b1c9/routeoverride-cni/0.log" Sep 15 12:11:53.424722 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.424698 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4mzd_4b6f665e-7431-4599-b9ef-302eb545b1c9/whereabouts-cni-bincopy/0.log" Sep 15 12:11:53.445041 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.445000 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4mzd_4b6f665e-7431-4599-b9ef-302eb545b1c9/whereabouts-cni/0.log" Sep 15 12:11:53.623246 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.623151 2573 scope.go:117] "RemoveContainer" containerID="dac3f3ad38aef219f1dc0041d7c9c1176f30691bcc91b764a10dc8f7f7d90f6b" Sep 15 12:11:53.633151 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.633130 2573 scope.go:117] "RemoveContainer" containerID="71f1d0130fd2492ec26c7c6cb3661078edba5a51743f7687b1e2467c015faf2e" Sep 15 12:11:53.646200 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.646158 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-c9qbq_9935a605-d4d3-4471-9740-cb993a66fe98/kube-multus/0.log" Sep 15 12:11:53.874584 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.874498 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ldmmr_91f95d2b-d44c-4f8a-83bf-649275a608e7/network-metrics-daemon/0.log" Sep 15 12:11:53.894044 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:53.894006 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ldmmr_91f95d2b-d44c-4f8a-83bf-649275a608e7/kube-rbac-proxy/0.log" Sep 15 12:11:55.015289 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:55.015257 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-controller/0.log" Sep 15 12:11:55.034111 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:55.034081 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/1.log" Sep 15 12:11:55.034281 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:55.034113 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/ovn-acl-logging/0.log" Sep 15 12:11:55.052817 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:55.052787 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/kube-rbac-proxy-node/0.log" Sep 15 12:11:55.075683 ip-10-0-128-189 kubenswrapper[2573]: I0915 12:11:55.075655 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kj4f2_087f48fe-aefc-4e62-b27f-d690c27cb027/kube-rbac-proxy-ovn-metrics/0.log"